2024-03-12T02:55:46.547 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-12T02:55:46.547 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --description orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} --archive /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854 --verbose --name teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854/orig.config.yaml 2024-03-12T02:55:46.577 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T02:55:46.664 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854 branch: main description: orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591854' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug ms: 1 global: mon election default strategy: 1 ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) mon_bind_msgr2: false sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: cephadm-package install: ceph: flavor: default sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b extra_packages: - cephadm selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c owner: scheduled_teuthology@teuthology priority: 200 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 6815 sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c targets: smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOZmR8x6IzoXKKffGo75V8AXIimC40GWHexbpHKPBZIRfJOBXq5tBOFFWc/69OPlPjzeNpehFVhuncEkmAYMDI= smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPTqnZGitYpcpb572mNbfZYAVXyeWniw3x8eD+/VyOZ7qdbAZyg3qwkHUkR1NHCpyNiYQwwNlxYvfCAWW6gGV54= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_20:08:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-12T02:55:46.664 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa; will attempt to use it 2024-03-12T02:55:46.665 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks 2024-03-12T02:55:46.665 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-12T02:55:46.670 INFO:teuthology.task.internal:Checking packages... 2024-03-12T02:55:46.695 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2b0101eb74d60610d544cb8a473e23a35ac1cc0b' 2024-03-12T02:55:46.695 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T02:55:46.695 INFO:teuthology.packaging:ref: None 2024-03-12T02:55:46.695 INFO:teuthology.packaging:tag: None 2024-03-12T02:55:46.696 INFO:teuthology.packaging:branch: main 2024-03-12T02:55:46.696 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T02:55:46.696 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-12T02:55:47.024 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1998.g1c349451 2024-03-12T02:55:47.025 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-12T02:55:47.034 INFO:teuthology.task.internal:no buildpackages task found 2024-03-12T02:55:47.034 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-12T02:55:47.051 INFO:teuthology.task.internal:Saving configuration 2024-03-12T02:55:47.062 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-12T02:55:47.072 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T02:55:47.098 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:50:43.752653', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOZmR8x6IzoXKKffGo75V8AXIimC40GWHexbpHKPBZIRfJOBXq5tBOFFWc/69OPlPjzeNpehFVhuncEkmAYMDI='} 2024-03-12T02:55:47.119 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:50:43.754495', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPTqnZGitYpcpb572mNbfZYAVXyeWniw3x8eD+/VyOZ7qdbAZyg3qwkHUkR1NHCpyNiYQwwNlxYvfCAWW6gGV54='} 2024-03-12T02:55:47.119 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-12T02:55:47.130 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-12T02:55:47.130 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-12T02:55:47.130 INFO:teuthology.run_tasks:Running task console_log... 2024-03-12T02:55:47.208 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff5b42fcf70>, signals=[15]) 2024-03-12T02:55:47.209 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-12T02:55:47.220 INFO:teuthology.task.internal:Opening connections... 2024-03-12T02:55:47.221 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2024-03-12T02:55:47.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:55:47.356 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-03-12T02:55:47.357 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:55:47.459 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-12T02:55:47.465 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2024-03-12T02:55:47.484 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2024-03-12T02:55:47.484 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2024-03-12T02:55:47.541 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2024-03-12T02:55:47.541 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="9" 2024-03-12T02:55:47.541 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2024-03-12T02:55:47.542 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2024-03-12T02:55:47.542 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="9" 2024-03-12T02:55:47.542 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el9" 2024-03-12T02:55:47.542 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T02:55:47.542 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2024-03-12T02:55:47.542 INFO:teuthology.orchestra.run.smithi094.stdout:LOGO="fedora-logo-icon" 2024-03-12T02:55:47.542 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T02:55:47.542 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2024-03-12T02:55:47.542 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T02:55:47.542 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T02:55:47.542 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T02:55:47.543 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2024-03-12T02:55:47.568 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-03-12T02:55:47.586 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-03-12T02:55:47.586 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-03-12T02:55:47.643 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2024-03-12T02:55:47.644 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="9" 2024-03-12T02:55:47.644 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2024-03-12T02:55:47.644 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2024-03-12T02:55:47.644 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="9" 2024-03-12T02:55:47.644 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el9" 2024-03-12T02:55:47.644 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T02:55:47.644 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2024-03-12T02:55:47.644 INFO:teuthology.orchestra.run.smithi151.stdout:LOGO="fedora-logo-icon" 2024-03-12T02:55:47.644 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T02:55:47.644 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2024-03-12T02:55:47.644 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T02:55:47.644 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T02:55:47.644 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T02:55:47.645 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-03-12T02:55:47.667 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-12T02:55:47.676 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-12T02:55:47.690 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-12T02:55:47.690 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2024-03-12T02:55:47.692 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-03-12T02:55:47.708 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-12T02:55:47.723 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-12T02:55:47.723 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2024-03-12T02:55:47.748 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-03-12T02:55:47.863 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-12T02:55:47.918 INFO:teuthology.run_tasks:Running task kernel... 2024-03-12T02:55:47.937 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-12T02:55:47.937 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-12T02:55:47.937 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T02:55:47.938 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T02:55:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:55:47.954 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-03-12T02:55:47.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:55:47.955 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-03-12T02:55:48.011 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-361.el9.x86_64 2024-03-12T02:55:48.011 INFO:teuthology.task.kernel:Running kernel on smithi094: 5.14.0-361.el9.x86_64 2024-03-12T02:55:48.011 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2024-03-12T02:55:48.012 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-361.el9.x86_64 2024-03-12T02:55:48.012 INFO:teuthology.task.kernel:Running kernel on smithi151: 5.14.0-361.el9.x86_64 2024-03-12T02:55:48.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum install -y kernel 2024-03-12T02:55:50.244 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-12T02:55:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-12T02:55:51.844 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-03-12T02:55:52.081 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-12T02:55:53.451 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 8.0 MB 00:01 2024-03-12T02:55:55.178 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-03-12T02:55:56.147 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-03-12T02:55:59.704 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 6.0 MB/s | 6.1 MB 00:01 2024-03-12T02:56:01.231 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2024-03-12T02:56:04.265 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 21 MB 00:02 2024-03-12T02:56:09.257 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-12T02:56:11.455 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T02:56:11.456 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T02:56:11.554 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T02:56:11.565 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T02:56:11.566 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-03-12T02:56:11.566 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T02:56:11.566 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T02:56:11.566 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T02:56:11.566 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-12T02:56:11.566 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T02:56:11.566 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T02:56:11.566 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T02:56:11.566 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T02:56:11.566 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T02:56:11.566 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T02:56:11.566 INFO:teuthology.orchestra.run.smithi094.stdout:Install 4 Packages 2024-03-12T02:56:11.566 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T02:56:11.567 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 100 M 2024-03-12T02:56:11.567 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 123 M 2024-03-12T02:56:11.567 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T02:56:12.892 INFO:teuthology.orchestra.run.smithi094.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 5.7 MB/s | 6.0 MB 00:01 2024-03-12T02:56:13.551 INFO:teuthology.orchestra.run.smithi094.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 12 MB/s | 21 MB 00:01 2024-03-12T02:56:14.609 INFO:teuthology.orchestra.run.smithi094.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 14 MB/s | 39 MB 00:02 2024-03-12T02:56:15.334 INFO:teuthology.orchestra.run.smithi094.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 14 MB/s | 33 MB 00:02 2024-03-12T02:56:15.334 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T02:56:15.335 INFO:teuthology.orchestra.run.smithi094.stdout:Total 26 MB/s | 100 MB 00:03 2024-03-12T02:56:16.217 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T02:56:16.613 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T02:56:16.613 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T02:56:17.956 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T02:56:17.957 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T02:56:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T02:56:20.869 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 816 kB/s | 19 MB 00:23 2024-03-12T02:56:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T02:56:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:56:24.248 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:56:24.431 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:56:29.208 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:56:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:56:34.295 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:56:34.319 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 634 kB/s | 6.1 MB 00:09 2024-03-12T02:56:35.761 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 15 kB 00:00 2024-03-12T02:56:37.232 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-12T02:56:42.105 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-12T02:56:44.303 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T02:56:44.303 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T02:56:44.410 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T02:56:44.426 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T02:56:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-03-12T02:56:44.426 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T02:56:44.426 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T02:56:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T02:56:44.426 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-12T02:56:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T02:56:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T02:56:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T02:56:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T02:56:44.427 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T02:56:44.427 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T02:56:44.427 INFO:teuthology.orchestra.run.smithi151.stdout:Install 4 Packages 2024-03-12T02:56:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T02:56:44.427 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 100 M 2024-03-12T02:56:44.427 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 123 M 2024-03-12T02:56:44.427 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T02:56:48.865 INFO:teuthology.orchestra.run.smithi151.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 1.4 MB/s | 6.0 MB 00:04 2024-03-12T02:57:00.663 INFO:teuthology.orchestra.run.smithi151.stdout:(2/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 2.4 MB/s | 39 MB 00:16 2024-03-12T02:57:03.088 INFO:teuthology.orchestra.run.smithi151.stdout:(3/4): kernel-core-5.14.0-427.el9.x86_64.rpm 1.1 MB/s | 21 MB 00:18 2024-03-12T02:57:03.329 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:57:03.329 INFO:teuthology.orchestra.run.smithi094.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T02:57:03.329 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T02:57:03.413 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:57:03.801 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:57:03.801 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T02:57:03.801 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:57:03.801 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:57:04.121 INFO:teuthology.orchestra.run.smithi151.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 2.2 MB/s | 33 MB 00:15 2024-03-12T02:57:04.121 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T02:57:04.121 INFO:teuthology.orchestra.run.smithi151.stdout:Total 5.1 MB/s | 100 MB 00:19 2024-03-12T02:57:04.213 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:57:04.214 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T02:57:04.214 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T02:57:04.214 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T02:57:04.214 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T02:57:04.214 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T02:57:04.214 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T02:57:04.214 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T02:57:04.214 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T02:57:04.767 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T02:57:04.907 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T02:57:04.907 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T02:57:04.907 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T02:57:04.908 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T02:57:04.908 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T02:57:04.908 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-12T02:57:04.908 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T02:57:04.908 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T02:57:05.007 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T02:57:05.029 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T02:57:05.030 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T02:57:05.030 DEBUG:teuthology.orchestra.run.smithi094:> sudo rpm -qi grub2-tools 2024-03-12T02:57:05.069 INFO:teuthology.orchestra.run.smithi094.stdout:Name : grub2-tools 2024-03-12T02:57:05.069 INFO:teuthology.orchestra.run.smithi094.stdout:Epoch : 1 2024-03-12T02:57:05.069 INFO:teuthology.orchestra.run.smithi094.stdout:Version : 2.06 2024-03-12T02:57:05.069 INFO:teuthology.orchestra.run.smithi094.stdout:Release : 68.el9 2024-03-12T02:57:05.069 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: x86_64 2024-03-12T02:57:05.069 INFO:teuthology.orchestra.run.smithi094.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T02:57:05.069 INFO:teuthology.orchestra.run.smithi094.stdout:Group : Unspecified 2024-03-12T02:57:05.070 INFO:teuthology.orchestra.run.smithi094.stdout:Size : 8264254 2024-03-12T02:57:05.070 INFO:teuthology.orchestra.run.smithi094.stdout:License : GPLv3+ 2024-03-12T02:57:05.070 INFO:teuthology.orchestra.run.smithi094.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T02:57:05.070 INFO:teuthology.orchestra.run.smithi094.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T02:57:05.070 INFO:teuthology.orchestra.run.smithi094.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T02:57:05.070 INFO:teuthology.orchestra.run.smithi094.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T02:57:05.070 INFO:teuthology.orchestra.run.smithi094.stdout:Packager : builder@centos.org 2024-03-12T02:57:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:Vendor : CentOS 2024-03-12T02:57:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T02:57:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:Summary : Support tools for GRUB. 2024-03-12T02:57:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:Description : 2024-03-12T02:57:05.071 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T02:57:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T02:57:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T02:57:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T02:57:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:hardware devices. 2024-03-12T02:57:05.071 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T02:57:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T02:57:05.072 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T02:57:05.072 INFO:teuthology.task.kernel:Updating grub on smithi094 to boot 5.14.0-427.el9.x86_64 2024-03-12T02:57:05.072 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T02:57:05.391 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T02:57:05.391 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T02:57:05.687 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2024-03-12T02:57:06.752 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T02:57:06.752 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T02:57:07.096 INFO:teuthology.orchestra.run.smithi094.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T02:57:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:done 2024-03-12T02:57:07.124 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-03-12T02:57:07.139 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.XjdvBivEcS 2024-03-12T02:57:07.139 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XjdvBivEcS 2024-03-12T02:57:07.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.XjdvBivEcS 2024-03-12T02:57:07.351 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.XjdvBivEcS is 6KB 2024-03-12T02:57:07.400 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.XjdvBivEcS 2024-03-12T02:57:07.413 DEBUG:teuthology.orchestra.run.smithi094:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T02:57:07.478 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T02:57:07.478 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T02:57:07.478 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T02:57:07.478 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T02:57:07.480 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T02:57:07.629 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2024-03-12T02:57:09.674 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T02:57:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T02:57:10.522 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:57:13.306 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:57:13.499 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:57:18.245 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:57:18.459 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:57:23.219 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:57:37.633 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-12T02:57:37.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-12T02:57:37.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:57:52.919 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:57:52.919 INFO:teuthology.orchestra.run.smithi151.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T02:57:52.919 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T02:57:53.008 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:57:53.468 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:57:53.468 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T02:57:53.468 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:57:53.469 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:57:53.892 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:57:53.893 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T02:57:53.893 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T02:57:53.893 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T02:57:53.893 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T02:57:53.893 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T02:57:53.893 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T02:57:53.893 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T02:57:53.893 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T02:57:54.464 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T02:57:54.652 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T02:57:54.653 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T02:57:54.653 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T02:57:54.653 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T02:57:54.653 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-12T02:57:54.653 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi151.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T02:57:54.653 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T02:57:54.774 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T02:57:54.775 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T02:57:54.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo rpm -qi grub2-tools 2024-03-12T02:57:54.815 INFO:teuthology.orchestra.run.smithi151.stdout:Name : grub2-tools 2024-03-12T02:57:54.815 INFO:teuthology.orchestra.run.smithi151.stdout:Epoch : 1 2024-03-12T02:57:54.815 INFO:teuthology.orchestra.run.smithi151.stdout:Version : 2.06 2024-03-12T02:57:54.815 INFO:teuthology.orchestra.run.smithi151.stdout:Release : 68.el9 2024-03-12T02:57:54.815 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: x86_64 2024-03-12T02:57:54.815 INFO:teuthology.orchestra.run.smithi151.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T02:57:54.815 INFO:teuthology.orchestra.run.smithi151.stdout:Group : Unspecified 2024-03-12T02:57:54.815 INFO:teuthology.orchestra.run.smithi151.stdout:Size : 8264254 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout:License : GPLv3+ 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout:Packager : builder@centos.org 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout:Vendor : CentOS 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout:Summary : Support tools for GRUB. 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout:Description : 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T02:57:54.816 INFO:teuthology.orchestra.run.smithi151.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T02:57:54.817 INFO:teuthology.orchestra.run.smithi151.stdout:hardware devices. 2024-03-12T02:57:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T02:57:54.817 INFO:teuthology.orchestra.run.smithi151.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T02:57:54.817 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T02:57:54.818 INFO:teuthology.task.kernel:Updating grub on smithi151 to boot 5.14.0-427.el9.x86_64 2024-03-12T02:57:54.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T02:57:55.450 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2024-03-12T02:57:55.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2024-03-12T02:57:56.841 INFO:teuthology.orchestra.run.smithi151.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T02:57:56.861 INFO:teuthology.orchestra.run.smithi151.stderr:done 2024-03-12T02:57:56.863 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-03-12T02:57:56.878 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.khy9iLccNu 2024-03-12T02:57:56.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.khy9iLccNu 2024-03-12T02:57:56.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.khy9iLccNu 2024-03-12T02:57:57.092 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.khy9iLccNu is 6KB 2024-03-12T02:57:57.140 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.khy9iLccNu 2024-03-12T02:57:57.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T02:57:57.219 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T02:57:57.219 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T02:57:57.220 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T02:57:57.220 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T02:57:57.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T02:57:57.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo shutdown -r now 2024-03-12T02:58:05.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-12T02:58:05.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:58:27.396 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-12T02:58:27.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-12T02:58:27.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:58:45.919 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-12T02:58:54.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-12T02:58:54.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:58:57.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-12T02:59:05.016 DEBUG:teuthology.orchestra.remote:timed out 2024-03-12T02:59:09.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-12T02:59:09.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:59:17.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-12T02:59:17.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:59:17.484 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-03-12T02:59:17.885 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-12T02:59:17.885 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-12T02:59:17.885 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-03-12T02:59:17.901 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-427.el9.x86_64 2024-03-12T02:59:17.901 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T02:59:17.902 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T02:59:17.902 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T02:59:18.902 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-12T02:59:18.902 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T02:59:18.990 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2024-03-12T02:59:19.024 DEBUG:teuthology.parallel:result is None 2024-03-12T02:59:28.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-12T02:59:43.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-12T02:59:43.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:59:43.839 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-03-12T02:59:44.270 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-12T02:59:44.271 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-12T02:59:44.271 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-03-12T02:59:44.285 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-427.el9.x86_64 2024-03-12T02:59:44.285 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T02:59:44.285 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T02:59:44.285 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T02:59:45.286 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-12T02:59:45.286 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T02:59:45.366 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2024-03-12T02:59:45.412 DEBUG:teuthology.parallel:result is None 2024-03-12T02:59:45.412 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-12T02:59:45.419 INFO:teuthology.task.internal:Creating test directory... 2024-03-12T02:59:45.420 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T02:59:45.422 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T02:59:45.437 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-12T02:59:45.443 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-12T02:59:45.453 INFO:teuthology.task.internal:Creating archive directory... 2024-03-12T02:59:45.454 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T02:59:45.481 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T02:59:45.566 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-12T02:59:45.573 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-12T02:59:45.573 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T02:59:45.575 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T02:59:45.607 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T02:59:45.612 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T02:59:45.620 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T02:59:45.626 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T02:59:45.628 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-12T02:59:45.635 INFO:teuthology.task.internal:Configuring sudo... 2024-03-12T02:59:45.635 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T02:59:45.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T02:59:45.697 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-12T02:59:45.706 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-12T02:59:45.706 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T02:59:45.740 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T02:59:45.754 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T02:59:45.837 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T02:59:45.885 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T02:59:45.967 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T02:59:46.011 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T02:59:46.011 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T02:59:46.078 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T02:59:46.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T02:59:46.161 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T02:59:46.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T02:59:46.278 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-12T02:59:46.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T02:59:46.342 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2024-03-12T02:59:46.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2024-03-12T02:59:46.385 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T02:59:46.429 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T02:59:46.900 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-12T02:59:46.908 INFO:teuthology.task.internal:Starting timer... 2024-03-12T02:59:46.908 INFO:teuthology.run_tasks:Running task pcp... 2024-03-12T02:59:46.920 INFO:teuthology.run_tasks:Running task selinux... 2024-03-12T02:59:46.928 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-12T02:59:46.928 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2024-03-12T02:59:46.990 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2024-03-12T02:59:46.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2024-03-12T02:59:47.051 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: 2024-03-12T02:59:47.053 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-12T02:59:47.053 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2024-03-12T02:59:47.080 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2024-03-12T02:59:47.080 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2024-03-12T02:59:47.097 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2024-03-12T02:59:47.097 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi094.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi151.front.sepia.ceph.com': 'permissive'} 2024-03-12T02:59:47.097 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T02:59:47.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:59:47.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T02:59:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:59:47.178 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-12T02:59:47.178 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2024-03-12T02:59:47.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2024-03-12T02:59:47.264 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-12T02:59:47.273 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-12T02:59:47.290 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-12T02:59:47.291 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi094.front.sepia.ceph.com,smithi151.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-12T03:03:56.837 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi094.front.sepia.ceph.com'), Remote(name='ubuntu@smithi151.front.sepia.ceph.com')] 2024-03-12T03:03:56.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-12T03:03:56.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:03:56.934 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-03-12T03:03:57.008 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-12T03:03:57.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-12T03:03:57.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:03:57.099 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-03-12T03:03:57.189 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-12T03:03:57.190 INFO:teuthology.run_tasks:Running task clock... 2024-03-12T03:03:57.201 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-12T03:03:57.201 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T03:03:57.201 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T03:03:57.204 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T03:03:57.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T03:03:57.233 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T03:03:57.250 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T03:03:57.260 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T03:03:57.277 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T03:03:57.279 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2024-03-12T03:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2024-03-12T03:03:57.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2024-03-12T03:03:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T03:03:57.320 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2024-03-12T03:03:57.325 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T03:03:57.337 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T03:03:57.353 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T03:03:57.376 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-03-12T03:03:57.379 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T03:03:57.379 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-03-12T03:03:57.379 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:03:57.379 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:03:57.380 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:03:57.380 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:03:57.412 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-03-12T03:03:57.415 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T03:03:57.415 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-03-12T03:03:57.416 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:03:57.416 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:03:57.416 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:03:57.416 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:03:57.417 INFO:teuthology.run_tasks:Running task install... 2024-03-12T03:03:57.424 DEBUG:teuthology.task.install:project ceph 2024-03-12T03:03:57.425 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}, 'extra_packages': ['cephadm']} 2024-03-12T03:03:57.425 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'} 2024-03-12T03:03:57.425 INFO:teuthology.task.install:Using flavor: default 2024-03-12T03:03:57.430 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-12T03:03:57.431 INFO:teuthology.task.install:extra packages: [] 2024-03-12T03:03:57.431 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'tag': None, 'wait_for_package': False} 2024-03-12T03:03:57.431 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:03:57.433 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'tag': None, 'wait_for_package': False} 2024-03-12T03:03:57.434 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:03:57.660 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/ 2024-03-12T03:03:57.660 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1979.g2b0101eb 2024-03-12T03:03:57.673 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/ 2024-03-12T03:03:57.673 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1979.g2b0101eb 2024-03-12T03:03:57.857 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-12T03:03:57.857 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:03:57.857 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-12T03:03:57.860 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-12T03:03:57.861 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-12T03:03:57.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-12T03:03:57.887 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-12T03:03:57.887 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-12T03:03:57.889 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-12T03:03:57.889 DEBUG:teuthology.orchestra.run.smithi151:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-12T03:03:57.956 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-12T03:03:57.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-12T03:03:58.037 DEBUG:teuthology.orchestra.run.smithi151:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-12T03:03:58.038 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-12T03:03:58.103 INFO:teuthology.orchestra.run.smithi151.stdout:check_obsoletes = 1 2024-03-12T03:03:58.104 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2024-03-12T03:03:58.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum clean all 2024-03-12T03:03:58.107 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2024-03-12T03:03:58.349 INFO:teuthology.orchestra.run.smithi094.stdout:74 files removed 2024-03-12T03:03:58.349 INFO:teuthology.orchestra.run.smithi151.stdout:74 files removed 2024-03-12T03:03:58.375 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw 2024-03-12T03:03:58.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw 2024-03-12T03:03:59.066 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 218 kB/s | 82 kB 00:00 2024-03-12T03:03:59.106 INFO:teuthology.orchestra.run.smithi151.stdout:ceph packages for x86_64 202 kB/s | 82 kB 00:00 2024-03-12T03:03:59.458 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-12T03:03:59.481 INFO:teuthology.orchestra.run.smithi151.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-12T03:03:59.875 INFO:teuthology.orchestra.run.smithi151.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-03-12T03:03:59.909 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-03-12T03:04:00.345 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-12T03:04:00.351 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-12T03:04:01.777 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-12T03:04:01.979 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-12T03:04:02.836 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-12T03:04:12.302 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 822 kB/s | 8.0 MB 00:09 2024-03-12T03:04:17.628 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 19 MB 00:04 2024-03-12T03:04:17.725 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:13 2024-03-12T03:04:21.695 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-12T03:04:23.254 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 87 kB/s | 15 kB 00:00 2024-03-12T03:04:25.308 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.1 MB 00:04 2024-03-12T03:04:26.892 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2024-03-12T03:04:27.719 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-03-12T03:04:29.064 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 3.6 MB/s | 21 MB 00:05 2024-03-12T03:04:32.644 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-12T03:04:33.905 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-12T03:04:35.206 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:04:35.219 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:04:35.219 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-12T03:04:35.219 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:04:35.219 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:04:35.219 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 9.8 M 2024-03-12T03:04:35.219 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrading: 2024-03-12T03:04:35.219 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.4 M 2024-03-12T03:04:35.220 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-12T03:04:35.220 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-12T03:04:35.220 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options 2024-03-12T03:04:35.220 INFO:teuthology.orchestra.run.smithi151.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-12T03:04:35.220 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.3 M 2024-03-12T03:04:35.220 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 20 M 2024-03-12T03:04:35.220 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 25 k 2024-03-12T03:04:35.220 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-12T03:04:35.220 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-12T03:04:35.220 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-12T03:04:35.220 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 745 k 2024-03-12T03:04:35.220 INFO:teuthology.orchestra.run.smithi151.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-12T03:04:35.220 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-12T03:04:35.221 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-12T03:04:35.221 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 501 k 2024-03-12T03:04:35.221 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-12T03:04:35.221 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.2 M 2024-03-12T03:04:35.221 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-12T03:04:35.221 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-12T03:04:35.221 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-12T03:04:35.221 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse 2024-03-12T03:04:35.221 INFO:teuthology.orchestra.run.smithi151.stdout: x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 45 k 2024-03-12T03:04:35.221 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 131 k 2024-03-12T03:04:35.221 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 162 k 2024-03-12T03:04:35.221 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-12T03:04:35.221 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 319 k 2024-03-12T03:04:35.222 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 298 k 2024-03-12T03:04:35.222 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 99 k 2024-03-12T03:04:35.222 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-12T03:04:35.222 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-12T03:04:35.222 INFO:teuthology.orchestra.run.smithi151.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-12T03:04:35.222 INFO:teuthology.orchestra.run.smithi151.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-12T03:04:35.222 INFO:teuthology.orchestra.run.smithi151.stdout:Installing weak dependencies: 2024-03-12T03:04:35.222 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-12T03:04:35.222 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:04:35.222 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:04:35.222 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:04:35.222 INFO:teuthology.orchestra.run.smithi151.stdout:Install 30 Packages 2024-03-12T03:04:35.223 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrade 2 Packages 2024-03-12T03:04:35.223 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:04:35.223 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 58 M 2024-03-12T03:04:35.223 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:04:36.328 INFO:teuthology.orchestra.run.smithi151.stdout:(1/32): ceph-radosgw-19.0.0-1979.g2b0101eb.el9. 13 MB/s | 9.8 MB 00:00 2024-03-12T03:04:36.370 INFO:teuthology.orchestra.run.smithi151.stdout:(2/32): ceph-selinux-19.0.0-1979.g2b0101eb.el9. 609 kB/s | 25 kB 00:00 2024-03-12T03:04:36.453 INFO:teuthology.orchestra.run.smithi151.stdout:(3/32): libcephfs2-19.0.0-1979.g2b0101eb.el9.x8 8.7 MB/s | 745 kB 00:00 2024-03-12T03:04:36.474 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:04:36.487 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:04:36.487 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-12T03:04:36.488 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:04:36.488 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:04:36.488 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 9.8 M 2024-03-12T03:04:36.488 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2024-03-12T03:04:36.488 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.4 M 2024-03-12T03:04:36.488 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-12T03:04:36.488 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-12T03:04:36.488 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options 2024-03-12T03:04:36.488 INFO:teuthology.orchestra.run.smithi094.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-12T03:04:36.488 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.3 M 2024-03-12T03:04:36.488 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 20 M 2024-03-12T03:04:36.488 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 25 k 2024-03-12T03:04:36.488 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 745 k 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 501 k 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.2 M 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse 2024-03-12T03:04:36.489 INFO:teuthology.orchestra.run.smithi094.stdout: x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 45 k 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 131 k 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 162 k 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 319 k 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 298 k 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 99 k 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout:Installing weak dependencies: 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:04:36.490 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:04:36.491 INFO:teuthology.orchestra.run.smithi094.stdout:Install 30 Packages 2024-03-12T03:04:36.491 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 2 Packages 2024-03-12T03:04:36.491 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:04:36.491 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 58 M 2024-03-12T03:04:36.491 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:04:36.512 INFO:teuthology.orchestra.run.smithi151.stdout:(4/32): libradosstriper1-19.0.0-1979.g2b0101eb. 8.4 MB/s | 501 kB 00:00 2024-03-12T03:04:36.829 INFO:teuthology.orchestra.run.smithi151.stdout:(5/32): librgw2-19.0.0-1979.g2b0101eb.el9.x86_6 16 MB/s | 5.2 MB 00:00 2024-03-12T03:04:36.871 INFO:teuthology.orchestra.run.smithi151.stdout:(6/32): python3-ceph-argparse-19.0.0-1979.g2b01 1.1 MB/s | 45 kB 00:00 2024-03-12T03:04:36.921 INFO:teuthology.orchestra.run.smithi151.stdout:(7/32): python3-ceph-common-19.0.0-1979.g2b0101 2.6 MB/s | 131 kB 00:00 2024-03-12T03:04:36.980 INFO:teuthology.orchestra.run.smithi151.stdout:(8/32): python3-cephfs-19.0.0-1979.g2b0101eb.el 2.7 MB/s | 162 kB 00:00 2024-03-12T03:04:37.047 INFO:teuthology.orchestra.run.smithi151.stdout:(9/32): ceph-base-19.0.0-1979.g2b0101eb.el9.x86 3.6 MB/s | 5.3 MB 00:01 2024-03-12T03:04:37.072 INFO:teuthology.orchestra.run.smithi151.stdout:(10/32): python3-rados-19.0.0-1979.g2b0101eb.el 3.4 MB/s | 319 kB 00:00 2024-03-12T03:04:37.122 INFO:teuthology.orchestra.run.smithi151.stdout:(11/32): python3-rbd-19.0.0-1979.g2b0101eb.el9. 3.9 MB/s | 298 kB 00:00 2024-03-12T03:04:37.147 INFO:teuthology.orchestra.run.smithi151.stdout:(12/32): python3-rgw-19.0.0-1979.g2b0101eb.el9. 1.3 MB/s | 99 kB 00:00 2024-03-12T03:04:37.281 INFO:teuthology.orchestra.run.smithi151.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 667 kB/s | 106 kB 00:00 2024-03-12T03:04:37.306 INFO:teuthology.orchestra.run.smithi151.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-12T03:04:37.324 INFO:teuthology.orchestra.run.smithi151.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-12T03:04:37.391 INFO:teuthology.orchestra.run.smithi151.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-12T03:04:37.416 INFO:teuthology.orchestra.run.smithi151.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-12T03:04:37.432 INFO:teuthology.orchestra.run.smithi151.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 995 kB/s | 41 kB 00:00 2024-03-12T03:04:37.516 INFO:teuthology.orchestra.run.smithi151.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 419 kB/s | 35 kB 00:00 2024-03-12T03:04:37.567 INFO:teuthology.orchestra.run.smithi151.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 888 kB/s | 44 kB 00:00 2024-03-12T03:04:37.592 INFO:teuthology.orchestra.run.smithi151.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.2 MB/s | 216 kB 00:00 2024-03-12T03:04:37.840 INFO:teuthology.orchestra.run.smithi094.stdout:(1/32): ceph-radosgw-19.0.0-1979.g2b0101eb.el9. 9.5 MB/s | 9.8 MB 00:01 2024-03-12T03:04:37.890 INFO:teuthology.orchestra.run.smithi094.stdout:(2/32): ceph-selinux-19.0.0-1979.g2b0101eb.el9. 504 kB/s | 25 kB 00:00 2024-03-12T03:04:37.974 INFO:teuthology.orchestra.run.smithi094.stdout:(3/32): libcephfs2-19.0.0-1979.g2b0101eb.el9.x8 8.7 MB/s | 745 kB 00:00 2024-03-12T03:04:38.084 INFO:teuthology.orchestra.run.smithi151.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 597 kB/s | 309 kB 00:00 2024-03-12T03:04:38.090 INFO:teuthology.orchestra.run.smithi094.stdout:(4/32): libradosstriper1-19.0.0-1979.g2b0101eb. 4.2 MB/s | 501 kB 00:00 2024-03-12T03:04:38.176 INFO:teuthology.orchestra.run.smithi151.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-12T03:04:38.276 INFO:teuthology.orchestra.run.smithi151.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 487 kB/s | 49 kB 00:00 2024-03-12T03:04:38.376 INFO:teuthology.orchestra.run.smithi151.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 673 kB/s | 67 kB 00:00 2024-03-12T03:04:38.485 INFO:teuthology.orchestra.run.smithi151.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 4.4 MB 00:00 2024-03-12T03:04:38.593 INFO:teuthology.orchestra.run.smithi151.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-12T03:04:38.641 INFO:teuthology.orchestra.run.smithi094.stdout:(5/32): librgw2-19.0.0-1979.g2b0101eb.el9.x86_6 9.4 MB/s | 5.2 MB 00:00 2024-03-12T03:04:38.677 INFO:teuthology.orchestra.run.smithi151.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 838 kB 00:00 2024-03-12T03:04:38.691 INFO:teuthology.orchestra.run.smithi094.stdout:(6/32): python3-ceph-argparse-19.0.0-1979.g2b01 910 kB/s | 45 kB 00:00 2024-03-12T03:04:38.766 INFO:teuthology.orchestra.run.smithi094.stdout:(7/32): python3-ceph-common-19.0.0-1979.g2b0101 1.7 MB/s | 131 kB 00:00 2024-03-12T03:04:38.816 INFO:teuthology.orchestra.run.smithi094.stdout:(8/32): python3-cephfs-19.0.0-1979.g2b0101eb.el 3.2 MB/s | 162 kB 00:00 2024-03-12T03:04:38.909 INFO:teuthology.orchestra.run.smithi094.stdout:(9/32): ceph-base-19.0.0-1979.g2b0101eb.el9.x86 2.5 MB/s | 5.3 MB 00:02 2024-03-12T03:04:38.934 INFO:teuthology.orchestra.run.smithi094.stdout:(10/32): python3-rados-19.0.0-1979.g2b0101eb.el 2.7 MB/s | 319 kB 00:00 2024-03-12T03:04:38.944 INFO:teuthology.orchestra.run.smithi151.stdout:(29/32): ceph-common-19.0.0-1979.g2b0101eb.el9. 5.9 MB/s | 20 MB 00:03 2024-03-12T03:04:38.959 INFO:teuthology.orchestra.run.smithi094.stdout:(11/32): python3-rbd-19.0.0-1979.g2b0101eb.el9. 5.8 MB/s | 298 kB 00:00 2024-03-12T03:04:38.984 INFO:teuthology.orchestra.run.smithi094.stdout:(12/32): python3-rgw-19.0.0-1979.g2b0101eb.el9. 1.9 MB/s | 99 kB 00:00 2024-03-12T03:04:39.135 INFO:teuthology.orchestra.run.smithi094.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 604 kB/s | 106 kB 00:00 2024-03-12T03:04:39.160 INFO:teuthology.orchestra.run.smithi094.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 908 kB/s | 159 kB 00:00 2024-03-12T03:04:39.177 INFO:teuthology.orchestra.run.smithi094.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-12T03:04:39.244 INFO:teuthology.orchestra.run.smithi094.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-12T03:04:39.269 INFO:teuthology.orchestra.run.smithi094.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-12T03:04:39.294 INFO:teuthology.orchestra.run.smithi094.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2024-03-12T03:04:39.311 INFO:teuthology.orchestra.run.smithi151.stdout:(30/32): librbd1-19.0.0-1979.g2b0101eb.el9.x86_ 8.6 MB/s | 3.1 MB 00:00 2024-03-12T03:04:39.395 INFO:teuthology.orchestra.run.smithi151.stdout:(31/32): librados2-19.0.0-1979.g2b0101eb.el9.x8 4.7 MB/s | 3.4 MB 00:00 2024-03-12T03:04:39.561 INFO:teuthology.orchestra.run.smithi094.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 131 kB/s | 35 kB 00:00 2024-03-12T03:04:39.620 INFO:teuthology.orchestra.run.smithi151.stdout:(32/32): thrift-0.15.0-2.el9.x86_64.rpm 1.5 MB/s | 1.6 MB 00:01 2024-03-12T03:04:39.623 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:04:39.623 INFO:teuthology.orchestra.run.smithi151.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-12T03:04:39.653 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:04:39.688 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:04:39.689 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:04:39.762 INFO:teuthology.orchestra.run.smithi094.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 222 kB/s | 44 kB 00:00 2024-03-12T03:04:39.828 INFO:teuthology.orchestra.run.smithi094.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 387 kB/s | 216 kB 00:00 2024-03-12T03:04:40.020 INFO:teuthology.orchestra.run.smithi094.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-12T03:04:40.079 INFO:teuthology.orchestra.run.smithi094.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 447 kB/s | 26 kB 00:00 2024-03-12T03:04:40.146 INFO:teuthology.orchestra.run.smithi094.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 729 kB/s | 49 kB 00:00 2024-03-12T03:04:40.170 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:04:40.171 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:04:40.204 INFO:teuthology.orchestra.run.smithi094.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-12T03:04:40.272 INFO:teuthology.orchestra.run.smithi094.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-12T03:04:40.338 INFO:teuthology.orchestra.run.smithi094.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-12T03:04:40.372 INFO:teuthology.orchestra.run.smithi094.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-03-12T03:04:40.430 INFO:teuthology.orchestra.run.smithi094.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-03-12T03:04:40.806 INFO:teuthology.orchestra.run.smithi094.stdout:(30/32): librbd1-19.0.0-1979.g2b0101eb.el9.x86_ 8.4 MB/s | 3.1 MB 00:00 2024-03-12T03:04:40.923 INFO:teuthology.orchestra.run.smithi094.stdout:(31/32): librados2-19.0.0-1979.g2b0101eb.el9.x8 6.1 MB/s | 3.4 MB 00:00 2024-03-12T03:04:41.644 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:04:41.665 INFO:teuthology.orchestra.run.smithi094.stdout:(32/32): ceph-common-19.0.0-1979.g2b0101eb.el9. 4.1 MB/s | 20 MB 00:04 2024-03-12T03:04:41.668 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:04:41.668 INFO:teuthology.orchestra.run.smithi094.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-12T03:04:41.700 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:04:41.738 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:04:41.738 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:04:41.763 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-12T03:04:41.799 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-12T03:04:41.832 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-12T03:04:41.856 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-12T03:04:41.873 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-12T03:04:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-12T03:04:42.078 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-12T03:04:42.224 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:04:42.224 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:04:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-12T03:04:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-12T03:04:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-12T03:04:42.708 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 8/34 2024-03-12T03:04:42.744 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-12T03:04:42.807 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-12T03:04:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-12T03:04:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 11/34 2024-03-12T03:04:42.970 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-12T03:04:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-12T03:04:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-12T03:04:43.070 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-12T03:04:43.182 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-12T03:04:43.221 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-12T03:04:43.262 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-12T03:04:43.295 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-12T03:04:43.323 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-12T03:04:43.356 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-12T03:04:43.683 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-12T03:04:43.719 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-12T03:04:43.753 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:04:43.808 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-12T03:04:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-12T03:04:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 23/34 2024-03-12T03:04:43.877 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-12T03:04:43.902 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-12T03:04:43.908 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-12T03:04:43.934 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-12T03:04:43.950 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-12T03:04:44.145 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-12T03:04:44.197 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-12T03:04:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-12T03:04:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-12T03:04:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-12T03:04:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-12T03:04:44.528 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 27/34 2024-03-12T03:04:44.557 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 28/34 2024-03-12T03:04:44.579 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-12T03:04:44.701 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-12T03:04:44.793 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 8/34 2024-03-12T03:04:44.814 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-12T03:04:44.860 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-12T03:04:44.889 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-12T03:04:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 11/34 2024-03-12T03:04:44.993 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-12T03:04:45.016 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-12T03:04:45.062 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-12T03:04:45.090 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-12T03:04:45.197 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-12T03:04:45.266 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-12T03:04:45.307 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-12T03:04:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-12T03:04:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-12T03:04:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-12T03:04:45.712 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-12T03:04:45.747 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-12T03:04:45.819 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-12T03:04:45.844 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 23/34 2024-03-12T03:04:45.886 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-12T03:04:46.352 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-12T03:04:46.383 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-12T03:04:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-12T03:04:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 27/34 2024-03-12T03:04:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 28/34 2024-03-12T03:04:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-12T03:04:46.745 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-12T03:04:47.306 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-12T03:04:47.340 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-12T03:04:47.423 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-12T03:04:47.423 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-12T03:04:47.423 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-12T03:04:47.423 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:04:47.466 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-12T03:04:48.685 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-12T03:04:48.731 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-12T03:04:49.282 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-12T03:04:49.376 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-12T03:04:49.519 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-12T03:04:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-12T03:04:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-12T03:04:49.520 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:04:49.561 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-12T03:04:57.092 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-12T03:04:57.092 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /sys 2024-03-12T03:04:57.092 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /proc 2024-03-12T03:04:57.093 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /mnt 2024-03-12T03:04:57.093 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /var/tmp 2024-03-12T03:04:57.093 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /home 2024-03-12T03:04:57.093 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /root 2024-03-12T03:04:57.093 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /tmp 2024-03-12T03:04:57.093 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:04:58.015 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-12T03:04:58.039 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-12T03:04:58.040 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:04:58.040 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-12T03:04:58.040 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T03:04:58.040 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T03:04:58.040 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:04:58.042 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-12T03:04:58.110 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-12T03:04:58.110 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-12T03:04:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-12T03:04:58.953 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2024-03-12T03:04:58.953 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2024-03-12T03:04:58.953 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2024-03-12T03:04:58.953 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2024-03-12T03:04:58.953 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2024-03-12T03:04:58.953 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /root 2024-03-12T03:04:58.954 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2024-03-12T03:04:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:04:59.278 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-12T03:04:59.278 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/34 2024-03-12T03:04:59.278 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/34 2024-03-12T03:04:59.278 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/34 2024-03-12T03:04:59.278 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/34 2024-03-12T03:04:59.278 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/34 2024-03-12T03:04:59.278 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 6/34 2024-03-12T03:04:59.278 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 8/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 9/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 10/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 11/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 13/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-12T03:04:59.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-12T03:04:59.280 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-12T03:04:59.280 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-12T03:04:59.280 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-12T03:04:59.280 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-12T03:04:59.280 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-12T03:04:59.280 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-12T03:04:59.280 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-12T03:04:59.280 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-12T03:04:59.280 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-12T03:04:59.281 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-12T03:04:59.281 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-12T03:04:59.281 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/34 2024-03-12T03:04:59.876 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-12T03:04:59.899 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-12T03:04:59.900 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:04:59.900 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-12T03:04:59.900 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T03:04:59.900 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T03:04:59.900 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:04:59.901 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-12T03:04:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-12T03:04:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-12T03:05:00.573 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-12T03:05:00.573 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:00.573 INFO:teuthology.orchestra.run.smithi151.stdout:Upgraded: 2024-03-12T03:05:00.573 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.573 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-12T03:05:00.574 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-12T03:05:00.575 INFO:teuthology.orchestra.run.smithi151.stdout: re2-1:20211101-3.el9.x86_64 2024-03-12T03:05:00.576 INFO:teuthology.orchestra.run.smithi151.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-12T03:05:00.576 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:00.576 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:05:00.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-test 2024-03-12T03:05:01.070 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-12T03:05:01.070 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/34 2024-03-12T03:05:01.070 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/34 2024-03-12T03:05:01.070 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/34 2024-03-12T03:05:01.070 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/34 2024-03-12T03:05:01.070 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/34 2024-03-12T03:05:01.070 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 6/34 2024-03-12T03:05:01.070 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-12T03:05:01.070 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 8/34 2024-03-12T03:05:01.070 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 9/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 10/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 11/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 13/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-12T03:05:01.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-12T03:05:01.072 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-12T03:05:01.072 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-12T03:05:01.072 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-12T03:05:01.072 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-12T03:05:01.072 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-12T03:05:01.072 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-12T03:05:01.072 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-12T03:05:01.073 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-12T03:05:01.073 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-12T03:05:01.073 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/34 2024-03-12T03:05:01.458 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:29 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:05:01.618 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:05:01.618 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:05:01.618 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 48 M 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 163 k 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout:Install 6 Packages 2024-03-12T03:05:01.619 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:01.620 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 48 M 2024-03-12T03:05:01.620 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 205 M 2024-03-12T03:05:01.620 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:05:01.884 INFO:teuthology.orchestra.run.smithi151.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-12T03:05:01.942 INFO:teuthology.orchestra.run.smithi151.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-12T03:05:01.976 INFO:teuthology.orchestra.run.smithi151.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-12T03:05:02.026 INFO:teuthology.orchestra.run.smithi151.stdout:(4/6): libcephsqlite-19.0.0-1979.g2b0101eb.el9. 474 kB/s | 163 kB 00:00 2024-03-12T03:05:02.164 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-12T03:05:02.164 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:02.164 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2024-03-12T03:05:02.164 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.165 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:05:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-12T03:05:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-12T03:05:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-12T03:05:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-12T03:05:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-12T03:05:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-12T03:05:02.166 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-12T03:05:02.166 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.166 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-12T03:05:02.166 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.166 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-12T03:05:02.166 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-12T03:05:02.166 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-12T03:05:02.166 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.166 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.166 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.166 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-12T03:05:02.166 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.166 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.167 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:02.167 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-12T03:05:02.167 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-12T03:05:02.167 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-12T03:05:02.167 INFO:teuthology.orchestra.run.smithi094.stdout: re2-1:20211101-3.el9.x86_64 2024-03-12T03:05:02.167 INFO:teuthology.orchestra.run.smithi094.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-12T03:05:02.167 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:02.167 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:05:02.452 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-test 2024-03-12T03:05:02.593 INFO:teuthology.orchestra.run.smithi151.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 303 kB/s | 187 kB 00:00 2024-03-12T03:05:03.005 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:30 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:05:03.159 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:05:03.160 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:05:03.160 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-12T03:05:03.160 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:05:03.160 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:05:03.160 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 48 M 2024-03-12T03:05:03.160 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-12T03:05:03.161 INFO:teuthology.orchestra.run.smithi094.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-12T03:05:03.161 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 163 k 2024-03-12T03:05:03.161 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-12T03:05:03.161 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-12T03:05:03.161 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-12T03:05:03.161 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:03.161 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:05:03.161 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:05:03.161 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-03-12T03:05:03.161 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:03.162 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 48 M 2024-03-12T03:05:03.162 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 205 M 2024-03-12T03:05:03.162 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:05:03.629 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-12T03:05:03.687 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-12T03:05:03.729 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-12T03:05:03.838 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): libcephsqlite-19.0.0-1979.g2b0101eb.el9. 401 kB/s | 163 kB 00:00 2024-03-12T03:05:04.010 INFO:teuthology.orchestra.run.smithi151.stdout:(6/6): ceph-test-19.0.0-1979.g2b0101eb.el9.x86_ 20 MB/s | 48 MB 00:02 2024-03-12T03:05:04.014 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:05:04.014 INFO:teuthology.orchestra.run.smithi151.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-12T03:05:04.029 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:05:04.050 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:05:04.051 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:05:04.435 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:05:04.436 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:05:04.680 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 197 kB/s | 187 kB 00:00 2024-03-12T03:05:05.019 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:05:05.094 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-12T03:05:05.143 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-12T03:05:05.186 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-12T03:05:05.228 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-12T03:05:05.256 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-12T03:05:05.914 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): ceph-test-19.0.0-1979.g2b0101eb.el9.x86_ 19 MB/s | 48 MB 00:02 2024-03-12T03:05:05.917 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:05:05.917 INFO:teuthology.orchestra.run.smithi094.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-12T03:05:05.931 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:05:05.953 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:05:05.953 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:05:06.340 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:05:06.341 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:05:06.910 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:05:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-12T03:05:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-12T03:05:07.079 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-12T03:05:07.122 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-12T03:05:07.150 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-12T03:05:09.805 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-12T03:05:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-12T03:05:11.633 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-12T03:05:11.708 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-12T03:05:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-12T03:05:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/6 2024-03-12T03:05:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/6 2024-03-12T03:05:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-12T03:05:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-12T03:05:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-12T03:05:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-12T03:05:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:12.885 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:05:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: jq-1.6-16.el9.x86_64 2024-03-12T03:05:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-12T03:05:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-12T03:05:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-12T03:05:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:12.886 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:05:13.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph 2024-03-12T03:05:13.760 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:41 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:05:13.920 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:05:13.922 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2024-03-12T03:05:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-12T03:05:13.923 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2024-03-12T03:05:13.923 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:05:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: ceph x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 6.6 k 2024-03-12T03:05:13.923 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-12T03:05:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 2.1 M 2024-03-12T03:05:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 1.5 M 2024-03-12T03:05:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 248 k 2024-03-12T03:05:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 4.7 M 2024-03-12T03:05:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 17 M 2024-03-12T03:05:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-12T03:05:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-12T03:05:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-12T03:05:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:13.926 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:05:13.926 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2024-03-12T03:05:13.926 INFO:teuthology.orchestra.run.smithi151.stdout:Install 36 Packages 2024-03-12T03:05:13.926 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:13.926 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 30 M 2024-03-12T03:05:13.926 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 107 M 2024-03-12T03:05:13.926 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:05:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-12T03:05:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/6 2024-03-12T03:05:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/6 2024-03-12T03:05:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-12T03:05:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-12T03:05:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-12T03:05:14.386 INFO:teuthology.orchestra.run.smithi151.stdout:(1/36): ceph-19.0.0-1979.g2b0101eb.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-03-12T03:05:14.502 INFO:teuthology.orchestra.run.smithi151.stdout:(2/36): ceph-mgr-19.0.0-1979.g2b0101eb.el9.x86_ 4.3 MB/s | 1.5 MB 00:00 2024-03-12T03:05:14.517 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-12T03:05:14.517 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:14.517 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:05:14.517 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:14.517 INFO:teuthology.orchestra.run.smithi094.stdout: jq-1.6-16.el9.x86_64 2024-03-12T03:05:14.518 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:14.518 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-12T03:05:14.518 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-12T03:05:14.518 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-12T03:05:14.518 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:14.518 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:05:14.545 INFO:teuthology.orchestra.run.smithi151.stdout:(3/36): ceph-mds-19.0.0-1979.g2b0101eb.el9.x86_ 5.4 MB/s | 2.1 MB 00:00 2024-03-12T03:05:14.638 INFO:teuthology.orchestra.run.smithi151.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1979.g2b01 2.6 MB/s | 248 kB 00:00 2024-03-12T03:05:14.787 INFO:teuthology.orchestra.run.smithi151.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-12T03:05:14.838 INFO:teuthology.orchestra.run.smithi151.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-12T03:05:14.851 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph 2024-03-12T03:05:14.879 INFO:teuthology.orchestra.run.smithi151.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 756 kB/s | 32 kB 00:00 2024-03-12T03:05:14.930 INFO:teuthology.orchestra.run.smithi151.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 937 kB/s | 47 kB 00:00 2024-03-12T03:05:14.972 INFO:teuthology.orchestra.run.smithi151.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 929 kB/s | 39 kB 00:00 2024-03-12T03:05:15.032 INFO:teuthology.orchestra.run.smithi151.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-12T03:05:15.115 INFO:teuthology.orchestra.run.smithi151.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-12T03:05:15.290 INFO:teuthology.orchestra.run.smithi151.stdout:(12/36): ceph-osd-19.0.0-1979.g2b0101eb.el9.x86 21 MB/s | 17 MB 00:00 2024-03-12T03:05:15.316 INFO:teuthology.orchestra.run.smithi151.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 513 kB/s | 103 kB 00:00 2024-03-12T03:05:15.341 INFO:teuthology.orchestra.run.smithi151.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-12T03:05:15.366 INFO:teuthology.orchestra.run.smithi151.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-12T03:05:15.438 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:42 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:05:15.492 INFO:teuthology.orchestra.run.smithi151.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 832 kB/s | 125 kB 00:00 2024-03-12T03:05:15.591 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:05:15.593 INFO:teuthology.orchestra.run.smithi094.stdout:========================================================================================= 2024-03-12T03:05:15.593 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-12T03:05:15.593 INFO:teuthology.orchestra.run.smithi094.stdout:========================================================================================= 2024-03-12T03:05:15.593 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:05:15.593 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 6.6 k 2024-03-12T03:05:15.593 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-12T03:05:15.593 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 2.1 M 2024-03-12T03:05:15.593 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 1.5 M 2024-03-12T03:05:15.594 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 248 k 2024-03-12T03:05:15.594 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 4.7 M 2024-03-12T03:05:15.594 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 17 M 2024-03-12T03:05:15.594 INFO:teuthology.orchestra.run.smithi094.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-12T03:05:15.594 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-12T03:05:15.594 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-12T03:05:15.594 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-12T03:05:15.594 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-12T03:05:15.594 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-12T03:05:15.594 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-12T03:05:15.594 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-12T03:05:15.594 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-12T03:05:15.595 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-12T03:05:15.596 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-12T03:05:15.596 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-12T03:05:15.596 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-12T03:05:15.596 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-12T03:05:15.596 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-12T03:05:15.596 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-12T03:05:15.596 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-12T03:05:15.596 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:15.596 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:05:15.596 INFO:teuthology.orchestra.run.smithi094.stdout:========================================================================================= 2024-03-12T03:05:15.596 INFO:teuthology.orchestra.run.smithi094.stdout:Install 36 Packages 2024-03-12T03:05:15.596 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:15.597 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 30 M 2024-03-12T03:05:15.597 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 107 M 2024-03-12T03:05:15.597 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:05:15.792 INFO:teuthology.orchestra.run.smithi151.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 503 kB/s | 151 kB 00:00 2024-03-12T03:05:15.867 INFO:teuthology.orchestra.run.smithi151.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 578 kB/s | 43 kB 00:00 2024-03-12T03:05:15.993 INFO:teuthology.orchestra.run.smithi151.stdout:(19/36): lua-devel-5.4.4-4.el9.x86_64.rpm 36 kB/s | 22 kB 00:00 2024-03-12T03:05:16.018 INFO:teuthology.orchestra.run.smithi151.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-03-12T03:05:16.094 INFO:teuthology.orchestra.run.smithi151.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-03-12T03:05:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:(22/36): ceph-mon-19.0.0-1979.g2b0101eb.el9.x86 2.6 MB/s | 4.7 MB 00:01 2024-03-12T03:05:16.186 INFO:teuthology.orchestra.run.smithi151.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-12T03:05:16.193 INFO:teuthology.orchestra.run.smithi094.stdout:(1/36): ceph-19.0.0-1979.g2b0101eb.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2024-03-12T03:05:16.270 INFO:teuthology.orchestra.run.smithi151.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-03-12T03:05:16.328 INFO:teuthology.orchestra.run.smithi151.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 138 kB/s | 23 kB 00:00 2024-03-12T03:05:16.354 INFO:teuthology.orchestra.run.smithi151.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2024-03-12T03:05:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:(27/36): python3-cherrypy-18.6.1-2.el9.noarch.r 910 kB/s | 358 kB 00:00 2024-03-12T03:05:16.438 INFO:teuthology.orchestra.run.smithi151.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 941 kB/s | 79 kB 00:00 2024-03-12T03:05:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:(29/36): python3-logutils-0.3.5-21.el9.noarch.r 344 kB/s | 46 kB 00:00 2024-03-12T03:05:16.514 INFO:teuthology.orchestra.run.smithi151.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-03-12T03:05:16.543 INFO:teuthology.orchestra.run.smithi094.stdout:(2/36): ceph-mgr-19.0.0-1979.g2b0101eb.el9.x86_ 2.6 MB/s | 1.5 MB 00:00 2024-03-12T03:05:16.589 INFO:teuthology.orchestra.run.smithi151.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 477 kB/s | 36 kB 00:00 2024-03-12T03:05:16.681 INFO:teuthology.orchestra.run.smithi151.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-12T03:05:16.715 INFO:teuthology.orchestra.run.smithi151.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 832 kB/s | 272 kB 00:00 2024-03-12T03:05:16.774 INFO:teuthology.orchestra.run.smithi151.stdout:(34/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 290 kB/s | 90 kB 00:00 2024-03-12T03:05:16.799 INFO:teuthology.orchestra.run.smithi151.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-03-12T03:05:16.824 INFO:teuthology.orchestra.run.smithi151.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-03-12T03:05:16.828 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:05:16.828 INFO:teuthology.orchestra.run.smithi151.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-12T03:05:16.851 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:05:16.896 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:05:16.896 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:05:17.043 INFO:teuthology.orchestra.run.smithi094.stdout:(3/36): ceph-mds-19.0.0-1979.g2b0101eb.el9.x86_ 2.0 MB/s | 2.1 MB 00:01 2024-03-12T03:05:17.152 INFO:teuthology.orchestra.run.smithi094.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1979.g2b01 2.2 MB/s | 248 kB 00:00 2024-03-12T03:05:17.302 INFO:teuthology.orchestra.run.smithi094.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-12T03:05:17.320 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:05:17.320 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:05:17.344 INFO:teuthology.orchestra.run.smithi094.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-12T03:05:17.386 INFO:teuthology.orchestra.run.smithi094.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 760 kB/s | 32 kB 00:00 2024-03-12T03:05:17.428 INFO:teuthology.orchestra.run.smithi094.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-12T03:05:17.470 INFO:teuthology.orchestra.run.smithi094.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 925 kB/s | 39 kB 00:00 2024-03-12T03:05:17.520 INFO:teuthology.orchestra.run.smithi094.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-12T03:05:17.604 INFO:teuthology.orchestra.run.smithi094.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-12T03:05:17.671 INFO:teuthology.orchestra.run.smithi094.stdout:(12/36): ceph-mon-19.0.0-1979.g2b0101eb.el9.x86 3.2 MB/s | 4.7 MB 00:01 2024-03-12T03:05:17.696 INFO:teuthology.orchestra.run.smithi094.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-12T03:05:17.713 INFO:teuthology.orchestra.run.smithi094.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-12T03:05:17.739 INFO:teuthology.orchestra.run.smithi094.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-12T03:05:17.922 INFO:teuthology.orchestra.run.smithi094.stdout:(16/36): ceph-osd-19.0.0-1979.g2b0101eb.el9.x86 12 MB/s | 17 MB 00:01 2024-03-12T03:05:17.973 INFO:teuthology.orchestra.run.smithi094.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 95 kB/s | 22 kB 00:00 2024-03-12T03:05:18.123 INFO:teuthology.orchestra.run.smithi094.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 305 kB/s | 125 kB 00:00 2024-03-12T03:05:18.671 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:05:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-12T03:05:18.786 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-12T03:05:18.813 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-12T03:05:18.892 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-12T03:05:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-12T03:05:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-12T03:05:19.062 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-12T03:05:19.143 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-12T03:05:19.191 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-12T03:05:19.272 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-12T03:05:19.334 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-12T03:05:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-12T03:05:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-12T03:05:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-12T03:05:19.713 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-12T03:05:19.760 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-12T03:05:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-12T03:05:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-12T03:05:19.856 INFO:teuthology.orchestra.run.smithi094.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 99 kB/s | 172 kB 00:01 2024-03-12T03:05:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-12T03:05:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-12T03:05:19.983 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-12T03:05:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-12T03:05:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-12T03:05:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-12T03:05:20.145 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-12T03:05:20.235 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-12T03:05:20.276 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-12T03:05:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-12T03:05:20.486 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-12T03:05:20.611 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-12T03:05:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 31/36 2024-03-12T03:05:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:21.290 INFO:teuthology.orchestra.run.smithi094.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 13 kB/s | 43 kB 00:03 2024-03-12T03:05:21.382 INFO:teuthology.orchestra.run.smithi094.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 235 kB/s | 358 kB 00:01 2024-03-12T03:05:21.441 INFO:teuthology.orchestra.run.smithi094.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 71 kB/s | 11 kB 00:00 2024-03-12T03:05:21.574 INFO:teuthology.orchestra.run.smithi094.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 174 kB/s | 23 kB 00:00 2024-03-12T03:05:21.591 INFO:teuthology.orchestra.run.smithi094.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 85 kB/s | 18 kB 00:00 2024-03-12T03:05:21.683 INFO:teuthology.orchestra.run.smithi094.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-03-12T03:05:21.858 INFO:teuthology.orchestra.run.smithi094.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 265 kB/s | 46 kB 00:00 2024-03-12T03:05:21.983 INFO:teuthology.orchestra.run.smithi094.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 50 kB/s | 20 kB 00:00 2024-03-12T03:05:22.159 INFO:teuthology.orchestra.run.smithi094.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 262 kB/s | 79 kB 00:00 2024-03-12T03:05:22.284 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-12T03:05:22.284 INFO:teuthology.orchestra.run.smithi094.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 131 kB/s | 16 kB 00:00 2024-03-12T03:05:22.309 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-12T03:05:22.309 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:05:22.309 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-12T03:05:22.309 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T03:05:22.310 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T03:05:22.310 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:22.751 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-12T03:05:22.774 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-12T03:05:22.774 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:05:22.774 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-12T03:05:22.774 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T03:05:22.774 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T03:05:22.774 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:22.859 INFO:teuthology.orchestra.run.smithi094.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 156 kB/s | 90 kB 00:00 2024-03-12T03:05:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-12T03:05:22.984 INFO:teuthology.orchestra.run.smithi094.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 272 kB/s | 272 kB 00:01 2024-03-12T03:05:23.099 INFO:teuthology.orchestra.run.smithi094.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 171 kB/s | 36 kB 00:00 2024-03-12T03:05:23.099 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-12T03:05:23.099 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:05:23.100 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-12T03:05:23.100 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T03:05:23.100 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T03:05:23.100 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:23.100 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-12T03:05:23.993 INFO:teuthology.orchestra.run.smithi094.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 228 kB/s | 230 kB 00:01 2024-03-12T03:05:24.135 INFO:teuthology.orchestra.run.smithi094.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 141 kB/s | 20 kB 00:00 2024-03-12T03:05:24.988 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-12T03:05:24.988 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/36 2024-03-12T03:05:24.988 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/36 2024-03-12T03:05:24.988 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/36 2024-03-12T03:05:24.988 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 6/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-12T03:05:24.989 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-12T03:05:24.990 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-12T03:05:24.990 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-12T03:05:24.990 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-12T03:05:24.990 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-12T03:05:24.990 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-12T03:05:24.990 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-12T03:05:24.990 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-12T03:05:24.990 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-12T03:05:24.990 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-12T03:05:24.990 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-12T03:05:24.990 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-12T03:05:24.990 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-12T03:05:24.990 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-12T03:05:24.991 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-12T03:05:24.991 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-12T03:05:24.991 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-12T03:05:24.991 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-12T03:05:25.127 INFO:teuthology.orchestra.run.smithi094.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 207 kB/s | 427 kB 00:02 2024-03-12T03:05:25.892 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-12T03:05:25.892 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:25.892 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: lua-5.4.4-4.el9.x86_64 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-12T03:05:25.893 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-12T03:05:25.894 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-12T03:05:25.894 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-12T03:05:25.894 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-12T03:05:25.894 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-12T03:05:25.894 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-12T03:05:25.894 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-12T03:05:25.894 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-12T03:05:25.894 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-12T03:05:25.894 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-12T03:05:25.894 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-12T03:05:25.894 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply-3.11-14.el9.noarch 2024-03-12T03:05:25.894 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-12T03:05:25.894 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-12T03:05:25.895 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-12T03:05:25.895 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-12T03:05:25.895 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-12T03:05:25.895 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-12T03:05:25.895 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-12T03:05:25.895 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-12T03:05:25.895 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-12T03:05:25.895 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-12T03:05:25.895 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-12T03:05:25.895 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:25.895 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:05:26.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-base 2024-03-12T03:05:26.819 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:54 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:05:26.916 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:05:26.966 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:05:26.968 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-12T03:05:26.968 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:05:27.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install cephadm 2024-03-12T03:05:27.595 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:55 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:05:27.742 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:05:27.743 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:05:27.743 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-12T03:05:27.743 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:05:27.743 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:05:27.743 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 748 k 2024-03-12T03:05:27.743 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:27.744 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:05:27.744 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:05:27.744 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-12T03:05:27.744 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:27.744 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 748 k 2024-03-12T03:05:27.744 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 755 k 2024-03-12T03:05:27.744 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:05:28.176 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm-19.0.0-1979.g2b0101eb.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-12T03:05:28.176 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:05:28.177 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-12T03:05:28.177 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:05:28.182 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:05:28.182 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:05:28.188 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:05:28.188 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:05:28.263 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:05:28.666 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:05:28.757 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:05:29.143 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:05:29.440 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:05:29.440 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:29.440 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:05:29.440 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:05:29.440 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:29.440 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:05:29.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-immutable-object-cache 2024-03-12T03:05:30.219 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:58 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:05:30.369 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:05:30.370 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:05:30.370 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-12T03:05:30.370 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:05:30.370 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:05:30.370 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 145 k 2024-03-12T03:05:30.370 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:30.370 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:05:30.370 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:05:30.370 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-12T03:05:30.370 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:30.370 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 145 k 2024-03-12T03:05:30.370 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 436 k 2024-03-12T03:05:30.371 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:05:30.720 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-immutable-object-cache-19.0.0-1979.g2b0101 416 kB/s | 145 kB 00:00 2024-03-12T03:05:30.720 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:05:30.720 INFO:teuthology.orchestra.run.smithi151.stdout:Total 414 kB/s | 145 kB 00:00 2024-03-12T03:05:30.720 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:05:30.728 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:05:30.728 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:05:30.781 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:05:30.782 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:05:30.917 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:05:30.971 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-12T03:05:30.995 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-12T03:05:30.995 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:05:30.995 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-12T03:05:30.995 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:32.007 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-12T03:05:32.008 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:32.008 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:05:32.008 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:32.008 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:32.008 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:05:32.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr 2024-03-12T03:05:32.838 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:00 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:05:32.935 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:05:32.988 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:05:32.989 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-12T03:05:32.989 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:05:33.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-dashboard 2024-03-12T03:05:33.633 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:01 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:05:33.784 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:05:33.785 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================== 2024-03-12T03:05:33.785 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-12T03:05:33.786 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================== 2024-03-12T03:05:33.786 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:05:33.786 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 1.8 M 2024-03-12T03:05:33.786 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-12T03:05:33.786 INFO:teuthology.orchestra.run.smithi151.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-12T03:05:33.786 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 34 k 2024-03-12T03:05:33.786 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 16 k 2024-03-12T03:05:33.786 INFO:teuthology.orchestra.run.smithi151.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-12T03:05:33.786 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-12T03:05:33.786 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-12T03:05:33.786 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-12T03:05:33.786 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-12T03:05:33.786 INFO:teuthology.orchestra.run.smithi151.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-12T03:05:33.787 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-12T03:05:33.787 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-12T03:05:33.787 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:33.787 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:05:33.787 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================== 2024-03-12T03:05:33.787 INFO:teuthology.orchestra.run.smithi151.stdout:Install 12 Packages 2024-03-12T03:05:33.787 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:33.787 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 6.9 M 2024-03-12T03:05:33.788 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 26 M 2024-03-12T03:05:33.788 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:05:34.611 INFO:teuthology.orchestra.run.smithi151.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1979.g2b0 84 kB/s | 16 kB 00:00 2024-03-12T03:05:34.711 INFO:teuthology.orchestra.run.smithi151.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1979.g2b 115 kB/s | 34 kB 00:00 2024-03-12T03:05:34.761 INFO:teuthology.orchestra.run.smithi151.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1979.g2b0101e 5.2 MB/s | 1.8 MB 00:00 2024-03-12T03:05:34.903 INFO:teuthology.orchestra.run.smithi151.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-03-12T03:05:34.929 INFO:teuthology.orchestra.run.smithi151.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 238 kB 00:00 2024-03-12T03:05:35.104 INFO:teuthology.orchestra.run.smithi151.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 111 kB/s | 19 kB 00:00 2024-03-12T03:05:35.280 INFO:teuthology.orchestra.run.smithi151.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.4 MB/s | 548 kB 00:00 2024-03-12T03:05:35.338 INFO:teuthology.orchestra.run.smithi151.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-03-12T03:05:35.372 INFO:teuthology.orchestra.run.smithi151.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 921 kB/s | 31 kB 00:00 2024-03-12T03:05:35.455 INFO:teuthology.orchestra.run.smithi151.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-12T03:05:35.822 INFO:teuthology.orchestra.run.smithi151.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.8 MB/s | 2.0 MB 00:00 2024-03-12T03:05:39.822 INFO:teuthology.orchestra.run.smithi151.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 171 kB/s | 863 kB 00:05 2024-03-12T03:05:39.827 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:05:39.828 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.1 MB/s | 6.9 MB 00:06 2024-03-12T03:05:39.847 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:05:39.867 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:05:39.867 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:05:40.057 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:05:40.057 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:05:40.572 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:05:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-12T03:05:40.692 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-12T03:05:40.739 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-12T03:05:40.769 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-12T03:05:40.875 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-12T03:05:41.004 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-12T03:05:41.101 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-12T03:05:41.144 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-12T03:05:41.181 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-12T03:05:41.214 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 10/12 2024-03-12T03:05:41.460 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 11/12 2024-03-12T03:05:41.497 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-12T03:05:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-12T03:05:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 1/12 2024-03-12T03:05:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 2/12 2024-03-12T03:05:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 3/12 2024-03-12T03:05:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-12T03:05:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-12T03:05:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-12T03:05:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-12T03:05:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-12T03:05:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-12T03:05:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-12T03:05:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-12T03:05:42.764 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-12T03:05:42.765 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:42.765 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:05:42.765 INFO:teuthology.orchestra.run.smithi151.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-12T03:05:42.765 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:05:42.765 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:05:42.765 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:05:42.765 INFO:teuthology.orchestra.run.smithi151.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-12T03:05:42.765 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-12T03:05:42.765 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-12T03:05:42.765 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-12T03:05:42.765 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-12T03:05:42.765 INFO:teuthology.orchestra.run.smithi151.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-12T03:05:42.765 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-12T03:05:42.766 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-12T03:05:42.766 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:42.766 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:05:43.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-12T03:05:43.587 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:11 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:05:43.748 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:05:43.749 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-03-12T03:05:43.749 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-12T03:05:43.749 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-03-12T03:05:43.749 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:05:43.749 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 7.4 M 2024-03-12T03:05:43.750 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-12T03:05:43.750 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-12T03:05:43.750 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-12T03:05:43.750 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-12T03:05:43.750 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-12T03:05:43.750 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-12T03:05:43.750 INFO:teuthology.orchestra.run.smithi151.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-12T03:05:43.750 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-12T03:05:43.750 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-12T03:05:43.750 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-12T03:05:43.750 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-12T03:05:43.750 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-12T03:05:43.751 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:43.751 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:05:43.751 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-03-12T03:05:43.751 INFO:teuthology.orchestra.run.smithi151.stdout:Install 12 Packages 2024-03-12T03:05:43.751 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:43.751 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 37 M 2024-03-12T03:05:43.751 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 209 M 2024-03-12T03:05:43.751 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:05:44.495 INFO:teuthology.orchestra.run.smithi151.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 170 kB/s | 32 kB 00:00 2024-03-12T03:05:44.528 INFO:teuthology.orchestra.run.smithi151.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 522 kB/s | 17 kB 00:00 2024-03-12T03:05:44.562 INFO:teuthology.orchestra.run.smithi151.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-12T03:05:44.621 INFO:teuthology.orchestra.run.smithi151.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.6 MB/s | 3.0 MB 00:00 2024-03-12T03:05:44.797 INFO:teuthology.orchestra.run.smithi151.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-12T03:05:44.872 INFO:teuthology.orchestra.run.smithi151.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-12T03:05:44.897 INFO:teuthology.orchestra.run.smithi151.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-12T03:05:45.014 INFO:teuthology.orchestra.run.smithi151.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-19 10 MB/s | 7.4 MB 00:00 2024-03-12T03:05:45.206 INFO:teuthology.orchestra.run.smithi151.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-12T03:05:45.373 INFO:teuthology.orchestra.run.smithi151.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 548 kB/s | 195 kB 00:00 2024-03-12T03:05:45.465 INFO:teuthology.orchestra.run.smithi151.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-12T03:05:46.224 INFO:teuthology.orchestra.run.smithi151.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 241 kB/s | 245 kB 00:01 2024-03-12T03:05:46.227 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:05:46.227 INFO:teuthology.orchestra.run.smithi151.stdout:Total 15 MB/s | 37 MB 00:02 2024-03-12T03:05:46.270 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:05:46.297 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:05:46.297 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:05:46.663 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:05:46.663 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:05:47.367 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:05:47.455 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-12T03:05:47.495 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-12T03:05:47.590 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-12T03:05:47.624 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-12T03:05:47.804 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-12T03:05:47.832 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-12T03:05:47.909 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-12T03:05:48.568 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-12T03:05:48.681 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-12T03:05:49.943 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-12T03:05:50.550 INFO:teuthology.orchestra.run.smithi094.stdout:(36/36): luarocks-3.9.2-1.el9.noarch.rpm 4.6 kB/s | 151 kB 00:32 2024-03-12T03:05:50.554 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:05:50.554 INFO:teuthology.orchestra.run.smithi094.stdout:Total 870 kB/s | 30 MB 00:34 2024-03-12T03:05:50.577 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:05:50.620 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:05:50.620 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:05:50.969 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-12T03:05:50.995 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-12T03:05:51.033 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:05:51.033 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:05:52.358 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:05:52.427 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-12T03:05:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-12T03:05:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-12T03:05:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-12T03:05:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-12T03:05:52.636 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-12T03:05:52.728 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-12T03:05:52.806 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-12T03:05:52.856 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-12T03:05:52.928 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-12T03:05:52.982 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-12T03:05:53.029 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-12T03:05:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-12T03:05:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-12T03:05:53.321 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-12T03:05:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-12T03:05:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-12T03:05:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-12T03:05:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-12T03:05:53.546 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-12T03:05:53.580 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-12T03:05:53.614 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-12T03:05:53.654 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-12T03:05:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-12T03:05:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 1/12 2024-03-12T03:05:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-12T03:05:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-12T03:05:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-12T03:05:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-12T03:05:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-12T03:05:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-12T03:05:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-12T03:05:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-12T03:05:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-12T03:05:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-12T03:05:53.689 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-12T03:05:53.724 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-12T03:05:53.812 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-12T03:05:53.846 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-12T03:05:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-12T03:05:54.055 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-12T03:05:54.083 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-12T03:05:54.083 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:54.083 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:05:54.084 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:05:54.084 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-12T03:05:54.084 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-12T03:05:54.084 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-12T03:05:54.084 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-12T03:05:54.084 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-12T03:05:54.084 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-12T03:05:54.084 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-12T03:05:54.084 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-12T03:05:54.084 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-12T03:05:54.084 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-12T03:05:54.084 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-12T03:05:54.084 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:54.085 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:05:54.180 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-12T03:05:54.363 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 31/36 2024-03-12T03:05:54.393 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-12T03:05:54.419 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-12T03:05:54.419 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:05:54.419 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-12T03:05:54.419 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T03:05:54.419 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T03:05:54.419 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:54.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-rook 2024-03-12T03:05:55.039 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:23 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:05:55.193 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:05:55.194 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-12T03:05:55.194 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-12T03:05:55.194 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-12T03:05:55.194 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:05:55.194 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 49 k 2024-03-12T03:05:55.194 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-12T03:05:55.194 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:05:55.195 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-12T03:05:55.196 INFO:teuthology.orchestra.run.smithi151.stdout:Install 13 Packages 2024-03-12T03:05:55.196 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:55.196 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 2.0 M 2024-03-12T03:05:55.196 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 26 M 2024-03-12T03:05:55.196 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:05:55.468 INFO:teuthology.orchestra.run.smithi151.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 199 kB/s | 24 kB 00:00 2024-03-12T03:05:55.485 INFO:teuthology.orchestra.run.smithi151.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 130 kB/s | 18 kB 00:00 2024-03-12T03:05:55.543 INFO:teuthology.orchestra.run.smithi151.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-12T03:05:55.569 INFO:teuthology.orchestra.run.smithi151.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-12T03:05:55.594 INFO:teuthology.orchestra.run.smithi151.stdout:(5/13): ceph-mgr-rook-19.0.0-1979.g2b0101eb.el9 198 kB/s | 49 kB 00:00 2024-03-12T03:05:55.620 INFO:teuthology.orchestra.run.smithi151.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-12T03:05:55.644 INFO:teuthology.orchestra.run.smithi151.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 570 kB/s | 43 kB 00:00 2024-03-12T03:05:55.820 INFO:teuthology.orchestra.run.smithi151.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 85 kB/s | 15 kB 00:00 2024-03-12T03:05:55.845 INFO:teuthology.orchestra.run.smithi151.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 128 kB/s | 32 kB 00:00 2024-03-12T03:05:55.912 INFO:teuthology.orchestra.run.smithi151.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-03-12T03:05:55.954 INFO:teuthology.orchestra.run.smithi151.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 655 kB/s | 203 kB 00:00 2024-03-12T03:05:55.980 INFO:teuthology.orchestra.run.smithi151.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-12T03:05:56.120 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-12T03:05:56.148 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-12T03:05:56.149 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:05:56.149 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-12T03:05:56.149 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T03:05:56.149 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T03:05:56.149 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:56.238 INFO:teuthology.orchestra.run.smithi151.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.4 MB/s | 1.0 MB 00:00 2024-03-12T03:05:56.243 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:05:56.243 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-12T03:05:56.243 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:05:56.256 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:05:56.256 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:05:56.359 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:05:56.360 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:05:56.596 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-12T03:05:56.619 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-12T03:05:56.619 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:05:56.619 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-12T03:05:56.619 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T03:05:56.619 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T03:05:56.619 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:56.822 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-12T03:05:56.844 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-12T03:05:56.845 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:05:56.845 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-12T03:05:56.845 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T03:05:56.845 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T03:05:56.845 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:56.883 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:05:56.903 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-12T03:05:57.053 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-12T03:05:57.114 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-12T03:05:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-12T03:05:57.233 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-12T03:05:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-12T03:05:57.368 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-12T03:05:57.471 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-12T03:05:57.527 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-12T03:05:58.252 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-12T03:05:58.308 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-12T03:05:58.342 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-12T03:05:58.390 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-12T03:05:58.413 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-12T03:05:58.552 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-12T03:05:58.552 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/36 2024-03-12T03:05:58.552 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/36 2024-03-12T03:05:58.552 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/36 2024-03-12T03:05:58.552 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/36 2024-03-12T03:05:58.552 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 6/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-12T03:05:58.553 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-12T03:05:58.554 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-12T03:05:58.554 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-12T03:05:58.554 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-12T03:05:58.554 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-12T03:05:58.554 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-12T03:05:58.554 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-12T03:05:58.554 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-12T03:05:58.554 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-12T03:05:58.555 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-12T03:05:58.555 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-12T03:05:58.555 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-12T03:05:58.555 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-12T03:05:58.555 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-12T03:05:58.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-12T03:05:58.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-12T03:05:58.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-12T03:05:59.212 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-12T03:05:59.212 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 1/13 2024-03-12T03:05:59.213 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-12T03:05:59.213 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-12T03:05:59.213 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-12T03:05:59.213 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-12T03:05:59.213 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-12T03:05:59.213 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-12T03:05:59.213 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-12T03:05:59.213 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-12T03:05:59.213 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-12T03:05:59.213 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-12T03:05:59.213 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-12T03:05:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-12T03:05:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:59.571 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:05:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:05:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:05:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: lua-5.4.4-4.el9.x86_64 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-12T03:05:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply-3.11-14.el9.noarch 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-12T03:05:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-12T03:05:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-12T03:05:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:05:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:05:59.587 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-12T03:05:59.587 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-12T03:05:59.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-12T03:05:59.589 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:05:59.589 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:05:59.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-cephadm 2024-03-12T03:05:59.961 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-base 2024-03-12T03:06:00.414 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:28 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:00.534 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:27 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:00.566 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:00.567 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-12T03:06:00.567 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-12T03:06:00.567 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-12T03:06:00.567 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:06:00.567 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 139 k 2024-03-12T03:06:00.567 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-12T03:06:00.567 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-12T03:06:00.567 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-12T03:06:00.567 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-12T03:06:00.568 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-12T03:06:00.568 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-12T03:06:00.568 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:00.568 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:06:00.568 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-12T03:06:00.568 INFO:teuthology.orchestra.run.smithi151.stdout:Install 6 Packages 2024-03-12T03:06:00.568 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:00.568 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 6.8 M 2024-03-12T03:06:00.569 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 33 M 2024-03-12T03:06:00.569 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:06:00.629 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:00.682 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:00.683 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-12T03:06:00.683 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:00.759 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install cephadm 2024-03-12T03:06:00.896 INFO:teuthology.orchestra.run.smithi151.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1979.g2b0101eb.e 627 kB/s | 139 kB 00:00 2024-03-12T03:06:00.980 INFO:teuthology.orchestra.run.smithi151.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-12T03:06:01.213 INFO:teuthology.orchestra.run.smithi151.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 460 kB/s | 248 kB 00:00 2024-03-12T03:06:01.238 INFO:teuthology.orchestra.run.smithi151.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 223 kB/s | 58 kB 00:00 2024-03-12T03:06:01.320 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:28 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:01.339 INFO:teuthology.orchestra.run.smithi151.stdout:(5/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-12T03:06:01.431 INFO:teuthology.orchestra.run.smithi151.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 235 kB/s | 51 kB 00:00 2024-03-12T03:06:01.434 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:01.434 INFO:teuthology.orchestra.run.smithi151.stdout:Total 7.8 MB/s | 6.8 MB 00:00 2024-03-12T03:06:01.446 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:06:01.457 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:06:01.457 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:06:01.466 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:01.467 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:01.467 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-12T03:06:01.467 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:01.467 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:06:01.467 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 748 k 2024-03-12T03:06:01.467 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:01.468 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:06:01.468 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:01.468 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-12T03:06:01.468 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:01.468 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 748 k 2024-03-12T03:06:01.468 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 755 k 2024-03-12T03:06:01.468 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:06:01.554 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:06:01.554 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:06:01.736 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm-19.0.0-1979.g2b0101eb.el9.noarch.rpm 2.7 MB/s | 748 kB 00:00 2024-03-12T03:06:01.737 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:01.737 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.7 MB/s | 748 kB 00:00 2024-03-12T03:06:01.737 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:06:01.742 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:06:01.742 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:06:01.748 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:06:01.748 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:06:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:06:01.857 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:06:02.028 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-12T03:06:02.074 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-12T03:06:02.261 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:06:02.332 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:06:02.592 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-12T03:06:02.677 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-12T03:06:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:06:02.740 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-12T03:06:02.763 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-12T03:06:02.970 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:06:02.970 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:02.971 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:06:02.971 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:06:02.971 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:02.971 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:03.233 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-immutable-object-cache 2024-03-12T03:06:03.603 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-12T03:06:03.603 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/6 2024-03-12T03:06:03.603 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-12T03:06:03.603 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-12T03:06:03.603 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-12T03:06:03.604 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-12T03:06:03.807 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:30 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:03.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-12T03:06:03.928 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:03.928 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:06:03.928 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:06:03.928 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-12T03:06:03.928 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-12T03:06:03.928 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-12T03:06:03.928 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-12T03:06:03.928 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-12T03:06:03.928 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:03.929 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:03.959 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:03.960 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-03-12T03:06:03.961 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:03.961 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:06:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 145 k 2024-03-12T03:06:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:03.961 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:06:03.961 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:03.961 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-12T03:06:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:03.962 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 145 k 2024-03-12T03:06:03.962 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 436 k 2024-03-12T03:06:03.962 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:06:04.152 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-immutable-object-cache-19.0.0-1979.g2b0101 761 kB/s | 145 kB 00:00 2024-03-12T03:06:04.153 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:04.153 INFO:teuthology.orchestra.run.smithi094.stdout:Total 755 kB/s | 145 kB 00:00 2024-03-12T03:06:04.153 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:06:04.160 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:06:04.160 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:06:04.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-fuse 2024-03-12T03:06:04.211 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:06:04.211 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:06:04.351 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:06:04.411 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-12T03:06:04.436 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-12T03:06:04.436 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:06:04.436 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-12T03:06:04.436 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:04.748 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:32 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:04.899 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:04.900 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-12T03:06:04.900 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:04.900 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:06:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 888 k 2024-03-12T03:06:04.900 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-12T03:06:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-12T03:06:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:04.900 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:06:04.900 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:04.900 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2024-03-12T03:06:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:04.901 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 966 k 2024-03-12T03:06:04.901 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.8 M 2024-03-12T03:06:04.901 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:06:05.044 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 552 kB/s | 79 kB 00:00 2024-03-12T03:06:05.211 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-fuse-19.0.0-1979.g2b0101eb.el9.x86_ 2.8 MB/s | 888 kB 00:00 2024-03-12T03:06:05.212 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:05.212 INFO:teuthology.orchestra.run.smithi151.stdout:Total 3.0 MB/s | 966 kB 00:00 2024-03-12T03:06:05.213 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:06:05.233 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:06:05.233 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:06:05.313 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:06:05.313 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:06:05.436 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-12T03:06:05.436 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:05.436 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:06:05.436 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:06:05.436 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:05.436 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:06:05.636 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-12T03:06:05.682 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr 2024-03-12T03:06:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-12T03:06:06.264 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:33 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:06.362 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:06.414 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:06.415 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-12T03:06:06.415 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:06.440 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-12T03:06:06.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/2 2024-03-12T03:06:06.478 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-dashboard 2024-03-12T03:06:06.754 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-12T03:06:06.755 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:06.755 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:06:06.755 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-12T03:06:06.755 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:06.755 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:07.034 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:34 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:07.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-volume 2024-03-12T03:06:07.183 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:07.184 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-03-12T03:06:07.184 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-12T03:06:07.184 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-03-12T03:06:07.185 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:06:07.185 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 1.8 M 2024-03-12T03:06:07.185 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-12T03:06:07.185 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-12T03:06:07.185 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 34 k 2024-03-12T03:06:07.185 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 16 k 2024-03-12T03:06:07.185 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-12T03:06:07.185 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-12T03:06:07.185 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-12T03:06:07.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-12T03:06:07.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-12T03:06:07.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-12T03:06:07.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-12T03:06:07.186 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-12T03:06:07.186 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:07.186 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:06:07.186 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-03-12T03:06:07.186 INFO:teuthology.orchestra.run.smithi094.stdout:Install 12 Packages 2024-03-12T03:06:07.186 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:07.187 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 6.9 M 2024-03-12T03:06:07.187 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 26 M 2024-03-12T03:06:07.187 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:06:07.557 INFO:teuthology.orchestra.run.smithi094.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1979.g2b 177 kB/s | 34 kB 00:00 2024-03-12T03:06:07.674 INFO:teuthology.orchestra.run.smithi094.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1979.g2b0 53 kB/s | 16 kB 00:00 2024-03-12T03:06:07.704 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:35 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:07.860 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:07.861 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:07.861 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-12T03:06:07.861 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:07.861 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:06:07.861 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 264 k 2024-03-12T03:06:07.861 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-12T03:06:07.861 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-12T03:06:07.861 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:07.861 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:06:07.862 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:07.862 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2024-03-12T03:06:07.862 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:07.862 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 332 k 2024-03-12T03:06:07.862 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 1.5 M 2024-03-12T03:06:07.862 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:06:07.916 INFO:teuthology.orchestra.run.smithi094.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1979.g2b0101e 3.3 MB/s | 1.8 MB 00:00 2024-03-12T03:06:07.958 INFO:teuthology.orchestra.run.smithi094.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-03-12T03:06:07.991 INFO:teuthology.orchestra.run.smithi094.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 751 kB/s | 238 kB 00:00 2024-03-12T03:06:08.003 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 488 kB/s | 69 kB 00:00 2024-03-12T03:06:08.075 INFO:teuthology.orchestra.run.smithi094.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 233 kB/s | 19 kB 00:00 2024-03-12T03:06:08.128 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-volume-19.0.0-1979.g2b0101eb.el9.no 992 kB/s | 264 kB 00:00 2024-03-12T03:06:08.129 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:08.129 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-12T03:06:08.130 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:06:08.134 INFO:teuthology.orchestra.run.smithi094.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.9 MB/s | 863 kB 00:00 2024-03-12T03:06:08.138 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:06:08.138 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:06:08.180 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:06:08.180 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:06:08.192 INFO:teuthology.orchestra.run.smithi094.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.3 MB/s | 548 kB 00:00 2024-03-12T03:06:08.243 INFO:teuthology.orchestra.run.smithi094.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 611 kB/s | 31 kB 00:00 2024-03-12T03:06:08.268 INFO:teuthology.orchestra.run.smithi094.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.1 MB/s | 144 kB 00:00 2024-03-12T03:06:08.310 INFO:teuthology.orchestra.run.smithi094.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 8.7 MB/s | 2.0 MB 00:00 2024-03-12T03:06:08.330 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:06:08.352 INFO:teuthology.orchestra.run.smithi094.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-03-12T03:06:08.355 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:08.355 INFO:teuthology.orchestra.run.smithi094.stdout:Total 5.9 MB/s | 6.9 MB 00:01 2024-03-12T03:06:08.372 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:06:08.390 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:06:08.390 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:06:08.511 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-12T03:06:08.556 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-12T03:06:08.570 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:06:08.570 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:06:08.572 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-12T03:06:08.572 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:06:08.573 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-12T03:06:08.573 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:09.090 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:06:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-12T03:06:09.196 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-12T03:06:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-12T03:06:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-12T03:06:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 1/2 2024-03-12T03:06:09.375 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-12T03:06:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-12T03:06:09.593 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-12T03:06:09.644 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-12T03:06:09.673 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-12T03:06:09.705 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 10/12 2024-03-12T03:06:09.755 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-12T03:06:09.755 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:09.755 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:06:09.756 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:06:09.756 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-12T03:06:09.756 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:09.756 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:09.948 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 11/12 2024-03-12T03:06:09.972 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-12T03:06:10.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados-devel 2024-03-12T03:06:10.683 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:38 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:10.818 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-12T03:06:10.818 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 1/12 2024-03-12T03:06:10.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 2/12 2024-03-12T03:06:10.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 3/12 2024-03-12T03:06:10.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-12T03:06:10.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-12T03:06:10.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-12T03:06:10.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-12T03:06:10.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-12T03:06:10.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-12T03:06:10.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-12T03:06:10.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-12T03:06:10.834 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:10.834 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:10.835 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-12T03:06:10.835 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:10.835 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:06:10.835 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 127 k 2024-03-12T03:06:10.835 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:10.835 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:06:10.835 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:10.835 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-12T03:06:10.835 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:10.836 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 127 k 2024-03-12T03:06:10.836 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 456 k 2024-03-12T03:06:10.836 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:06:11.054 INFO:teuthology.orchestra.run.smithi151.stdout:librados-devel-19.0.0-1979.g2b0101eb.el9.x86_64 581 kB/s | 127 kB 00:00 2024-03-12T03:06:11.055 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:11.055 INFO:teuthology.orchestra.run.smithi151.stdout:Total 577 kB/s | 127 kB 00:00 2024-03-12T03:06:11.055 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:06:11.061 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:06:11.061 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:06:11.117 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:06:11.117 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:06:11.181 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-12T03:06:11.181 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:11.181 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:06:11.181 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-12T03:06:11.181 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:06:11.181 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:06:11.181 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:06:11.181 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-12T03:06:11.181 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-12T03:06:11.181 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-12T03:06:11.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-12T03:06:11.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-12T03:06:11.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-12T03:06:11.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-12T03:06:11.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-12T03:06:11.182 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:11.182 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:11.273 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:06:11.379 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:11.446 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-12T03:06:11.822 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:11.998 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:38 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:12.126 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:12.126 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:12.126 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:06:12.126 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:06:12.126 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:12.126 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:12.149 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:12.150 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-03-12T03:06:12.150 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-12T03:06:12.150 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-03-12T03:06:12.150 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:06:12.150 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 7.4 M 2024-03-12T03:06:12.150 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-12T03:06:12.150 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout:Install 12 Packages 2024-03-12T03:06:12.151 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:12.152 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 37 M 2024-03-12T03:06:12.152 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 209 M 2024-03-12T03:06:12.152 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:06:12.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs2 2024-03-12T03:06:12.468 INFO:teuthology.orchestra.run.smithi094.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 223 kB/s | 32 kB 00:00 2024-03-12T03:06:12.509 INFO:teuthology.orchestra.run.smithi094.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 424 kB/s | 17 kB 00:00 2024-03-12T03:06:12.552 INFO:teuthology.orchestra.run.smithi094.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 845 kB/s | 34 kB 00:00 2024-03-12T03:06:12.610 INFO:teuthology.orchestra.run.smithi094.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-12T03:06:12.777 INFO:teuthology.orchestra.run.smithi094.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-12T03:06:12.844 INFO:teuthology.orchestra.run.smithi094.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-12T03:06:12.869 INFO:teuthology.orchestra.run.smithi094.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-12T03:06:12.980 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:40 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:13.080 INFO:teuthology.orchestra.run.smithi151.stdout:Package libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:13.103 INFO:teuthology.orchestra.run.smithi094.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 61 MB/s | 16 MB 00:00 2024-03-12T03:06:13.135 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:13.136 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-12T03:06:13.136 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:13.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs-devel 2024-03-12T03:06:13.337 INFO:teuthology.orchestra.run.smithi094.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-19 7.3 MB/s | 7.4 MB 00:01 2024-03-12T03:06:13.362 INFO:teuthology.orchestra.run.smithi094.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 755 kB/s | 195 kB 00:00 2024-03-12T03:06:13.496 INFO:teuthology.orchestra.run.smithi094.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-12T03:06:13.787 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:41 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:13.949 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:13.950 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-12T03:06:13.950 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:13.950 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:06:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 31 k 2024-03-12T03:06:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:13.950 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:06:13.950 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:13.950 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-12T03:06:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:13.950 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 31 k 2024-03-12T03:06:13.950 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 140 k 2024-03-12T03:06:13.951 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:06:14.121 INFO:teuthology.orchestra.run.smithi151.stdout:libcephfs-devel-19.0.0-1979.g2b0101eb.el9.x86_6 184 kB/s | 31 kB 00:00 2024-03-12T03:06:14.121 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:14.122 INFO:teuthology.orchestra.run.smithi151.stdout:Total 182 kB/s | 31 kB 00:00 2024-03-12T03:06:14.122 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:06:14.125 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:06:14.125 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:06:14.153 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:06:14.153 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:06:14.281 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:06:14.385 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:14.797 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:14.879 INFO:teuthology.orchestra.run.smithi094.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 400 kB/s | 803 kB 00:02 2024-03-12T03:06:14.881 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:14.881 INFO:teuthology.orchestra.run.smithi094.stdout:Total 14 MB/s | 37 MB 00:02 2024-03-12T03:06:14.924 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:06:14.950 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:06:14.951 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:06:15.074 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:15.074 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:15.074 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:06:15.074 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:06:15.074 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:15.074 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:15.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados2 2024-03-12T03:06:15.311 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:06:15.311 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:06:15.831 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:43 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:15.928 INFO:teuthology.orchestra.run.smithi151.stdout:Package librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:15.980 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:15.981 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-12T03:06:15.981 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:16.018 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:06:16.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librbd1 2024-03-12T03:06:16.085 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-12T03:06:16.125 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-12T03:06:16.217 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-12T03:06:16.262 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-12T03:06:16.438 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-12T03:06:16.469 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-12T03:06:16.545 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-12T03:06:16.624 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:44 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:16.722 INFO:teuthology.orchestra.run.smithi151.stdout:Package librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:16.773 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:16.775 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-12T03:06:16.775 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:16.833 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rados 2024-03-12T03:06:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-12T03:06:17.307 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-12T03:06:17.399 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:45 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:17.496 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:17.547 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:17.548 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-12T03:06:17.548 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:17.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rgw 2024-03-12T03:06:18.159 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:46 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:18.256 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:18.307 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:18.308 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-12T03:06:18.308 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:18.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-cephfs 2024-03-12T03:06:18.549 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-12T03:06:18.925 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:46 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:19.023 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:19.075 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:19.075 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-12T03:06:19.075 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:19.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rbd 2024-03-12T03:06:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-12T03:06:19.611 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-12T03:06:19.694 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:47 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:19.792 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:19.843 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:19.844 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-12T03:06:19.844 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:19.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-fuse 2024-03-12T03:06:20.463 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:48 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:20.610 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:20.610 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:20.610 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-03-12T03:06:20.611 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:20.611 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:06:20.611 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 85 k 2024-03-12T03:06:20.611 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:20.611 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:06:20.611 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:20.611 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-12T03:06:20.611 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:20.611 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 85 k 2024-03-12T03:06:20.611 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 231 k 2024-03-12T03:06:20.611 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:06:20.925 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-fuse-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 272 kB/s | 85 kB 00:00 2024-03-12T03:06:20.925 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:20.925 INFO:teuthology.orchestra.run.smithi151.stdout:Total 271 kB/s | 85 kB 00:00 2024-03-12T03:06:20.926 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:06:20.933 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:06:20.934 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:06:20.988 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:06:20.988 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:06:21.124 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:06:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:21.701 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:22.022 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:06:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:06:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:22.022 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-12T03:06:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 1/12 2024-03-12T03:06:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-12T03:06:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-12T03:06:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-12T03:06:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-12T03:06:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-12T03:06:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-12T03:06:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-12T03:06:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-12T03:06:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-12T03:06:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-12T03:06:22.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-mirror 2024-03-12T03:06:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-12T03:06:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:22.681 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:06:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:06:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-12T03:06:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-12T03:06:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-12T03:06:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-12T03:06:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-12T03:06:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-12T03:06:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-12T03:06:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-12T03:06:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-12T03:06:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-12T03:06:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-12T03:06:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:22.682 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:22.853 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:50 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:23.006 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:23.007 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-12T03:06:23.007 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:23.007 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:06:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-12T03:06:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:23.007 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:06:23.007 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:23.007 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-12T03:06:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:23.008 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 3.1 M 2024-03-12T03:06:23.008 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 13 M 2024-03-12T03:06:23.008 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:06:23.013 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-rook 2024-03-12T03:06:23.406 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-mirror-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 7.8 MB/s | 3.1 MB 00:00 2024-03-12T03:06:23.407 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:23.407 INFO:teuthology.orchestra.run.smithi151.stdout:Total 7.7 MB/s | 3.1 MB 00:00 2024-03-12T03:06:23.407 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:06:23.416 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:06:23.416 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:06:23.489 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:06:23.489 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:06:23.581 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:50 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:23.735 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:23.736 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-12T03:06:23.737 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-12T03:06:23.737 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-12T03:06:23.737 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:06:23.737 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 49 k 2024-03-12T03:06:23.737 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-12T03:06:23.737 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-12T03:06:23.737 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-12T03:06:23.737 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-12T03:06:23.737 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-12T03:06:23.737 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-12T03:06:23.737 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-12T03:06:23.737 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-12T03:06:23.738 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-12T03:06:23.738 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-12T03:06:23.738 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-12T03:06:23.738 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-12T03:06:23.738 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-12T03:06:23.738 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:23.738 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:06:23.738 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-12T03:06:23.738 INFO:teuthology.orchestra.run.smithi094.stdout:Install 13 Packages 2024-03-12T03:06:23.738 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:23.739 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 2.0 M 2024-03-12T03:06:23.739 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 26 M 2024-03-12T03:06:23.739 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:06:23.914 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:06:23.948 INFO:teuthology.orchestra.run.smithi094.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 184 kB/s | 24 kB 00:00 2024-03-12T03:06:23.973 INFO:teuthology.orchestra.run.smithi094.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2024-03-12T03:06:23.982 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:24.006 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:24.006 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:06:24.006 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-12T03:06:24.007 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T03:06:24.007 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T03:06:24.007 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:24.031 INFO:teuthology.orchestra.run.smithi094.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-12T03:06:24.056 INFO:teuthology.orchestra.run.smithi094.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-12T03:06:24.082 INFO:teuthology.orchestra.run.smithi094.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-12T03:06:24.107 INFO:teuthology.orchestra.run.smithi094.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-03-12T03:06:24.132 INFO:teuthology.orchestra.run.smithi094.stdout:(7/13): ceph-mgr-rook-19.0.0-1979.g2b0101eb.el9 155 kB/s | 49 kB 00:00 2024-03-12T03:06:24.224 INFO:teuthology.orchestra.run.smithi094.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-12T03:06:24.249 INFO:teuthology.orchestra.run.smithi094.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-12T03:06:24.350 INFO:teuthology.orchestra.run.smithi094.stdout:(10/13): python3-google-auth-2.28.0-1.el9.noarc 932 kB/s | 203 kB 00:00 2024-03-12T03:06:24.375 INFO:teuthology.orchestra.run.smithi094.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 471 kB/s | 59 kB 00:00 2024-03-12T03:06:24.409 INFO:teuthology.orchestra.run.smithi094.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-12T03:06:24.501 INFO:teuthology.orchestra.run.smithi094.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.7 MB/s | 1.0 MB 00:00 2024-03-12T03:06:24.721 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:24.721 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-03-12T03:06:24.721 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:06:24.721 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:06:24.721 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:06:24.721 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:06:24.722 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:06:25.129 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:06:25.181 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:25.181 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:25.181 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:06:25.182 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:06:25.182 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:25.182 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:25.295 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-12T03:06:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-12T03:06:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-12T03:06:25.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-nbd 2024-03-12T03:06:25.445 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-12T03:06:25.486 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-12T03:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-12T03:06:25.689 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-12T03:06:25.738 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-12T03:06:26.018 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:54 ago on Tue 12 Mar 2024 03:04:32 AM UTC. 2024-03-12T03:06:26.168 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-12T03:06:26.168 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:26.168 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-03-12T03:06:26.168 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:26.168 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-12T03:06:26.168 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 170 k 2024-03-12T03:06:26.168 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:26.168 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-12T03:06:26.169 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-12T03:06:26.169 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-12T03:06:26.169 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:26.169 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 170 k 2024-03-12T03:06:26.170 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 490 k 2024-03-12T03:06:26.170 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-12T03:06:26.405 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-nbd-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 725 kB/s | 170 kB 00:00 2024-03-12T03:06:26.406 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:26.406 INFO:teuthology.orchestra.run.smithi151.stdout:Total 720 kB/s | 170 kB 00:00 2024-03-12T03:06:26.406 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-12T03:06:26.413 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-12T03:06:26.413 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-12T03:06:26.459 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-12T03:06:26.470 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-12T03:06:26.471 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-12T03:06:26.510 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-12T03:06:26.544 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-12T03:06:26.592 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-12T03:06:26.614 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-12T03:06:26.624 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-12T03:06:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-12T03:06:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 1/13 2024-03-12T03:06:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-12T03:06:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-12T03:06:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-12T03:06:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-12T03:06:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-12T03:06:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-12T03:06:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-12T03:06:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-12T03:06:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-12T03:06:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-12T03:06:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-12T03:06:27.532 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:27.532 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:27.532 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-12T03:06:27.532 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:06:27.532 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:06:27.532 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-12T03:06:27.788 DEBUG:teuthology.parallel:result is None 2024-03-12T03:06:27.847 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-12T03:06:27.848 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-12T03:06:27.849 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-12T03:06:27.849 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:27.849 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:28.109 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-cephadm 2024-03-12T03:06:28.735 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:55 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:28.908 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:28.909 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-12T03:06:28.909 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-12T03:06:28.909 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-12T03:06:28.909 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:06:28.909 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 139 k 2024-03-12T03:06:28.909 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-12T03:06:28.910 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-12T03:06:28.910 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-12T03:06:28.910 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-12T03:06:28.910 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-12T03:06:28.910 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-12T03:06:28.910 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:28.910 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:06:28.910 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-12T03:06:28.910 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-03-12T03:06:28.910 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:28.911 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 6.8 M 2024-03-12T03:06:28.911 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 33 M 2024-03-12T03:06:28.911 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:06:29.355 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-12T03:06:29.371 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1979.g2b0101eb.e 463 kB/s | 139 kB 00:00 2024-03-12T03:06:29.397 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 763 kB/s | 248 kB 00:00 2024-03-12T03:06:29.888 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 112 kB/s | 58 kB 00:00 2024-03-12T03:06:29.914 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 99 kB/s | 51 kB 00:00 2024-03-12T03:06:30.147 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 691 kB/s | 548 kB 00:00 2024-03-12T03:06:30.150 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:30.150 INFO:teuthology.orchestra.run.smithi094.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-03-12T03:06:30.163 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:06:30.173 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:06:30.173 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:06:30.266 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:06:30.266 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:06:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:06:30.719 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-12T03:06:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-12T03:06:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-12T03:06:31.377 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-12T03:06:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-12T03:06:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-12T03:06:32.304 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-12T03:06:32.304 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/6 2024-03-12T03:06:32.304 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-12T03:06:32.304 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-12T03:06:32.304 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-12T03:06:32.304 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-12T03:06:32.605 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-12T03:06:32.606 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:32.606 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:06:32.606 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:06:32.606 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-12T03:06:32.606 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-12T03:06:32.606 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-12T03:06:32.606 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-12T03:06:32.606 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-12T03:06:32.606 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:32.606 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:32.852 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-fuse 2024-03-12T03:06:33.414 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:00 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:33.566 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:33.566 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:33.566 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-12T03:06:33.566 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:33.566 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:06:33.566 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 888 k 2024-03-12T03:06:33.566 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-12T03:06:33.567 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-12T03:06:33.567 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:33.567 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:06:33.567 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:33.567 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-03-12T03:06:33.567 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:33.567 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 966 k 2024-03-12T03:06:33.568 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 2.8 M 2024-03-12T03:06:33.568 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:06:33.708 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 562 kB/s | 79 kB 00:00 2024-03-12T03:06:33.991 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-fuse-19.0.0-1979.g2b0101eb.el9.x86_ 2.0 MB/s | 888 kB 00:00 2024-03-12T03:06:33.993 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:33.993 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.2 MB/s | 966 kB 00:00 2024-03-12T03:06:33.993 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:06:34.014 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:06:34.014 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:06:34.090 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:06:34.090 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:06:34.287 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:06:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-12T03:06:34.491 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-12T03:06:35.157 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-12T03:06:35.157 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/2 2024-03-12T03:06:35.472 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-12T03:06:35.473 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:35.473 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:06:35.473 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-12T03:06:35.473 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:35.473 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:35.762 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-volume 2024-03-12T03:06:36.344 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:03 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:36.491 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:36.492 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:36.492 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-12T03:06:36.492 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:36.492 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:06:36.492 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 264 k 2024-03-12T03:06:36.492 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-12T03:06:36.492 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-12T03:06:36.493 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:36.493 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:06:36.493 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:36.493 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-03-12T03:06:36.493 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:36.493 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 332 k 2024-03-12T03:06:36.493 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 1.5 M 2024-03-12T03:06:36.494 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:06:36.630 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 504 kB/s | 69 kB 00:00 2024-03-12T03:06:36.747 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-volume-19.0.0-1979.g2b0101eb.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-12T03:06:36.748 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:36.749 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-12T03:06:36.749 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:06:36.757 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:06:36.758 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:06:36.798 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:06:36.798 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:06:36.945 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:06:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-12T03:06:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-12T03:06:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-12T03:06:37.182 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:06:37.182 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-12T03:06:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:37.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 1/2 2024-03-12T03:06:38.261 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-12T03:06:38.262 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:38.262 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:06:38.262 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:06:38.262 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-12T03:06:38.262 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:38.263 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:38.637 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados-devel 2024-03-12T03:06:39.206 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:06 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:39.373 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:39.373 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:39.373 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-03-12T03:06:39.373 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:39.373 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:06:39.373 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 127 k 2024-03-12T03:06:39.374 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:39.374 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:06:39.374 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:39.374 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-12T03:06:39.374 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:39.375 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 127 k 2024-03-12T03:06:39.375 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 456 k 2024-03-12T03:06:39.375 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:06:39.687 INFO:teuthology.orchestra.run.smithi094.stdout:librados-devel-19.0.0-1979.g2b0101eb.el9.x86_64 405 kB/s | 127 kB 00:00 2024-03-12T03:06:39.688 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:39.688 INFO:teuthology.orchestra.run.smithi094.stdout:Total 403 kB/s | 127 kB 00:00 2024-03-12T03:06:39.688 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:06:39.693 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:06:39.694 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:06:39.748 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:06:39.749 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:06:39.889 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:06:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:40.462 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:40.779 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:40.779 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:40.779 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:06:40.779 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:06:40.779 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:40.779 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:41.043 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs2 2024-03-12T03:06:41.619 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:08 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:41.719 INFO:teuthology.orchestra.run.smithi094.stdout:Package libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:41.772 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:41.773 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-12T03:06:41.773 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:41.850 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs-devel 2024-03-12T03:06:42.410 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:09 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:42.561 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:42.561 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-03-12T03:06:42.561 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:42.562 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:06:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 31 k 2024-03-12T03:06:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:42.562 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:06:42.562 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:42.562 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-12T03:06:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:42.563 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 31 k 2024-03-12T03:06:42.563 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 140 k 2024-03-12T03:06:42.563 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:06:42.724 INFO:teuthology.orchestra.run.smithi094.stdout:libcephfs-devel-19.0.0-1979.g2b0101eb.el9.x86_6 193 kB/s | 31 kB 00:00 2024-03-12T03:06:42.725 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:42.725 INFO:teuthology.orchestra.run.smithi094.stdout:Total 191 kB/s | 31 kB 00:00 2024-03-12T03:06:42.725 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:06:42.729 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:06:42.729 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:06:42.748 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:06:42.749 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:06:42.878 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:06:42.972 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:43.663 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:43.663 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:43.663 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:06:43.663 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:06:43.663 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:43.663 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:43.835 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados2 2024-03-12T03:06:44.394 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:11 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:44.490 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:44.542 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:44.543 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-12T03:06:44.543 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:44.587 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librbd1 2024-03-12T03:06:45.140 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:12 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:45.237 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:45.288 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:45.289 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-12T03:06:45.289 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:45.335 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rados 2024-03-12T03:06:45.903 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:12 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:46.000 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:46.050 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:46.051 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-12T03:06:46.051 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:46.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rgw 2024-03-12T03:06:46.662 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:13 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:46.759 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:46.811 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:46.812 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-12T03:06:46.812 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:46.858 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-cephfs 2024-03-12T03:06:47.415 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:14 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:47.513 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:47.564 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:47.565 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-12T03:06:47.565 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:47.618 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rbd 2024-03-12T03:06:48.187 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:15 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:48.282 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:06:48.333 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:48.334 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-12T03:06:48.334 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:48.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-fuse 2024-03-12T03:06:48.938 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:15 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:49.087 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:49.087 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:49.088 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-03-12T03:06:49.088 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:49.088 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:06:49.088 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 85 k 2024-03-12T03:06:49.088 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:49.088 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:06:49.088 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:49.088 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-12T03:06:49.088 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:49.088 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 85 k 2024-03-12T03:06:49.088 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 231 k 2024-03-12T03:06:49.088 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:06:49.303 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-fuse-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 397 kB/s | 85 kB 00:00 2024-03-12T03:06:49.304 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:49.304 INFO:teuthology.orchestra.run.smithi094.stdout:Total 395 kB/s | 85 kB 00:00 2024-03-12T03:06:49.304 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:06:49.311 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:06:49.311 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:06:49.366 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:06:49.366 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:06:49.514 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:06:49.609 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:49.993 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:50.303 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:50.303 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:50.303 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:06:50.303 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:06:50.303 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:50.303 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:50.529 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-mirror 2024-03-12T03:06:51.101 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:18 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:51.254 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:51.255 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-03-12T03:06:51.255 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:51.255 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:06:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-12T03:06:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:51.255 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:06:51.256 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:51.256 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-12T03:06:51.256 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:51.256 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.1 M 2024-03-12T03:06:51.256 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 13 M 2024-03-12T03:06:51.256 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:06:51.824 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-mirror-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 5.4 MB/s | 3.1 MB 00:00 2024-03-12T03:06:51.824 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:51.824 INFO:teuthology.orchestra.run.smithi094.stdout:Total 5.4 MB/s | 3.1 MB 00:00 2024-03-12T03:06:51.825 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:06:51.833 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:06:51.834 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:06:51.906 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:06:51.906 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:06:52.326 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:06:52.391 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:52.414 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:52.415 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:06:52.415 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-12T03:06:52.415 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T03:06:52.415 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T03:06:52.415 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:53.602 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:53.602 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:53.602 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:06:53.602 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:06:53.602 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:53.602 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:53.872 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-nbd 2024-03-12T03:06:54.442 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:21 ago on Tue 12 Mar 2024 03:04:33 AM UTC. 2024-03-12T03:06:54.590 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-12T03:06:54.590 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:54.591 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-03-12T03:06:54.591 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:54.591 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-12T03:06:54.591 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 170 k 2024-03-12T03:06:54.591 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:54.591 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-12T03:06:54.591 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-12T03:06:54.591 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-12T03:06:54.591 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:54.591 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 170 k 2024-03-12T03:06:54.591 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 490 k 2024-03-12T03:06:54.591 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-12T03:06:54.936 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-nbd-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 494 kB/s | 170 kB 00:00 2024-03-12T03:06:54.937 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:06:54.937 INFO:teuthology.orchestra.run.smithi094.stdout:Total 492 kB/s | 170 kB 00:00 2024-03-12T03:06:54.937 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-12T03:06:54.945 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-12T03:06:54.945 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-12T03:06:55.001 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-12T03:06:55.001 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-12T03:06:55.146 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-12T03:06:55.259 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:56.036 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:06:56.036 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:56.036 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-12T03:06:56.036 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:06:56.036 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:06:56.036 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-12T03:06:56.306 DEBUG:teuthology.parallel:result is None 2024-03-12T03:06:56.307 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:06:56.450 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-12T03:06:56.471 INFO:teuthology.orchestra.run.smithi094.stdout:19.0.0-1979.g2b0101eb.el9 2024-03-12T03:06:56.471 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979.g2b0101eb.el9 2024-03-12T03:06:56.471 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979.g2b0101eb is installed. 2024-03-12T03:06:56.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:06:56.687 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-12T03:06:56.712 INFO:teuthology.orchestra.run.smithi151.stdout:19.0.0-1979.g2b0101eb.el9 2024-03-12T03:06:56.712 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979.g2b0101eb.el9 2024-03-12T03:06:56.713 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979.g2b0101eb is installed. 2024-03-12T03:06:56.714 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-12T03:06:56.715 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:06:56.715 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-12T03:06:56.743 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-12T03:06:56.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-12T03:06:56.784 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-12T03:06:56.784 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:06:56.785 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2024-03-12T03:06:56.811 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-12T03:06:56.878 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-12T03:06:56.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2024-03-12T03:06:56.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-12T03:06:56.979 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-12T03:06:56.979 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:06:56.979 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-12T03:06:57.006 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-12T03:06:57.074 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-12T03:06:57.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-12T03:06:57.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-12T03:06:57.172 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-12T03:06:57.172 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:06:57.173 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2024-03-12T03:06:57.199 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-12T03:06:57.266 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-12T03:06:57.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2024-03-12T03:06:57.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-12T03:06:57.360 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-12T03:06:57.459 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'mon_bind_msgr2': False, 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'cephadm_mode': 'cephadm-package'} 2024-03-12T03:06:57.459 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:06:57.459 INFO:tasks.cephadm:Cluster fsid is 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:06:57.459 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-12T03:06:57.460 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.94:6789]', 'mon.c': '[v1:172.21.15.94:6790]', 'mon.b': '[v1:172.21.15.151:6789]'} 2024-03-12T03:06:57.460 INFO:tasks.cephadm:First mon is mon.a on smithi094 2024-03-12T03:06:57.460 INFO:tasks.cephadm:First mgr is y 2024-03-12T03:06:57.460 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-12T03:06:57.460 DEBUG:teuthology.orchestra.run.smithi094:> sudo hostname $(hostname -s) 2024-03-12T03:06:57.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo hostname $(hostname -s) 2024-03-12T03:06:57.525 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b on all hosts... 2024-03-12T03:06:57.525 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-12T03:06:57.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-12T03:06:58.424 INFO:teuthology.orchestra.run.smithi151.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T03:06:58.439 INFO:teuthology.orchestra.run.smithi094.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T03:07:41.728 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-12T03:07:41.728 INFO:teuthology.orchestra.run.smithi094.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-12T03:07:41.728 INFO:teuthology.orchestra.run.smithi094.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-12T03:07:41.729 INFO:teuthology.orchestra.run.smithi094.stdout: "repo_digests": [ 2024-03-12T03:07:41.729 INFO:teuthology.orchestra.run.smithi094.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-12T03:07:41.729 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-03-12T03:07:41.729 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-12T03:07:52.004 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-03-12T03:07:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-12T03:07:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-12T03:07:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: "repo_digests": [ 2024-03-12T03:07:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-12T03:07:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-03-12T03:07:52.005 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-03-12T03:07:52.028 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph 2024-03-12T03:07:52.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph 2024-03-12T03:07:52.093 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /etc/ceph 2024-03-12T03:07:52.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /etc/ceph 2024-03-12T03:07:52.160 INFO:tasks.cephadm:Writing seed config... 2024-03-12T03:07:52.162 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-12T03:07:52.162 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-12T03:07:52.162 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-12T03:07:52.162 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-12T03:07:52.162 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-03-12T03:07:52.162 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-12T03:07:52.162 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-12T03:07:52.162 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-12T03:07:52.162 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-12T03:07:52.163 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-03-12T03:07:52.163 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-12T03:07:52.163 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-12T03:07:52.163 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-03-12T03:07:52.163 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-03-12T03:07:52.163 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-12T03:07:52.163 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:07:52.164 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-12T03:07:52.191 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 964142e8-e01d-11ee-95c8-87774f69a715 mon election default strategy = 1 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-03-12T03:07:52.192 DEBUG:teuthology.orchestra.run.smithi094:mon.a> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.a.service 2024-03-12T03:07:52.234 DEBUG:teuthology.orchestra.run.smithi094:mgr.y> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@mgr.y.service 2024-03-12T03:07:52.278 INFO:tasks.cephadm:Bootstrapping... 2024-03-12T03:07:52.278 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b -v bootstrap --fsid 964142e8-e01d-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.94:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:07:52.471 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:07:52.472 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b', '-v', 'bootstrap', '--fsid', '964142e8-e01d-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.94:6789]', '--skip-admin-label'] 2024-03-12T03:07:52.497 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T03:07:52.498 INFO:teuthology.orchestra.run.smithi094.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-12T03:07:52.498 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying podman|docker is present... 2024-03-12T03:07:52.521 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T03:07:52.522 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying lvm2 is present... 2024-03-12T03:07:52.522 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying time synchronization is in place... 2024-03-12T03:07:52.529 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T03:07:52.529 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T03:07:52.534 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T03:07:52.535 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-03-12T03:07:52.541 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout enabled 2024-03-12T03:07:52.547 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout active 2024-03-12T03:07:52.547 INFO:teuthology.orchestra.run.smithi094.stdout:Unit chronyd.service is enabled and running 2024-03-12T03:07:52.547 INFO:teuthology.orchestra.run.smithi094.stdout:Repeating the final host check... 2024-03-12T03:07:52.569 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T03:07:52.570 INFO:teuthology.orchestra.run.smithi094.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-12T03:07:52.570 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl is present 2024-03-12T03:07:52.570 INFO:teuthology.orchestra.run.smithi094.stdout:lvcreate is present 2024-03-12T03:07:52.576 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T03:07:52.576 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T03:07:52.582 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T03:07:52.582 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-03-12T03:07:52.588 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout enabled 2024-03-12T03:07:52.594 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout active 2024-03-12T03:07:52.594 INFO:teuthology.orchestra.run.smithi094.stdout:Unit chronyd.service is enabled and running 2024-03-12T03:07:52.594 INFO:teuthology.orchestra.run.smithi094.stdout:Host looks OK 2024-03-12T03:07:52.594 INFO:teuthology.orchestra.run.smithi094.stdout:Cluster fsid: 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:07:52.594 INFO:teuthology.orchestra.run.smithi094.stdout:Acquiring lock 139883779104192 on /run/cephadm/964142e8-e01d-11ee-95c8-87774f69a715.lock 2024-03-12T03:07:52.594 INFO:teuthology.orchestra.run.smithi094.stdout:Lock 139883779104192 acquired on /run/cephadm/964142e8-e01d-11ee-95c8-87774f69a715.lock 2024-03-12T03:07:52.595 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying IP 172.21.15.94 port 6789 ... 2024-03-12T03:07:52.595 INFO:teuthology.orchestra.run.smithi094.stdout:Base mon IP(s) is [172.21.15.94:6789], mon addrv is [v1:172.21.15.94:6789] 2024-03-12T03:07:52.598 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.94 metric 100 2024-03-12T03:07:52.598 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.94 metric 100 2024-03-12T03:07:52.600 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-12T03:07:52.601 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-12T03:07:52.601 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2024-03-12T03:07:52.603 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-12T03:07:52.603 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-12T03:07:52.603 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T03:07:52.603 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-12T03:07:52.603 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d763/64 scope link 2024-03-12T03:07:52.603 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T03:07:52.604 INFO:teuthology.orchestra.run.smithi094.stdout:Mon IP `172.21.15.94` is in CIDR network `172.21.0.0/20` 2024-03-12T03:07:52.604 INFO:teuthology.orchestra.run.smithi094.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-03-12T03:07:52.604 INFO:teuthology.orchestra.run.smithi094.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-12T03:07:52.605 INFO:teuthology.orchestra.run.smithi094.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T03:07:53.060 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stdout 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-12T03:07:53.060 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T03:07:53.060 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Getting image source signatures 2024-03-12T03:07:53.060 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Copying blob sha256:4392e0adce526b0883d65bbedc13478c05c0e708702f1c7015b93fe77c162cd5 2024-03-12T03:07:53.060 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-12T03:07:53.060 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Copying config sha256:4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-12T03:07:53.060 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-12T03:07:54.499 INFO:teuthology.orchestra.run.smithi094.stdout:ceph: stdout ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-12T03:07:54.499 INFO:teuthology.orchestra.run.smithi094.stdout:Ceph version: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-12T03:07:54.499 INFO:teuthology.orchestra.run.smithi094.stdout:Extracting ceph user uid/gid from container image... 2024-03-12T03:07:55.379 INFO:teuthology.orchestra.run.smithi094.stdout:stat: stdout 167 167 2024-03-12T03:07:55.379 INFO:teuthology.orchestra.run.smithi094.stdout:Creating initial keys... 2024-03-12T03:08:02.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph-authtool: stdout AQALx+9l55fZLhAAj37DfFIjUDdjdD4gUtLdaQ== 2024-03-12T03:08:03.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph-authtool: stdout AQASx+9lXXTEMhAA+2jmh0VhqnFqihC4J5lNWw== 2024-03-12T03:08:04.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph-authtool: stdout AQATx+9lA8OvLhAAml2jwjVEMljJ7yFKZVyHyA== 2024-03-12T03:08:04.373 INFO:teuthology.orchestra.run.smithi094.stdout:Creating initial monmap... 2024-03-12T03:08:05.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T03:08:05.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-12T03:08:05.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:08:05.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T03:08:05.263 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool for a [v1:172.21.15.94:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T03:08:05.263 INFO:teuthology.orchestra.run.smithi094.stdout:setting min_mon_release = quincy 2024-03-12T03:08:05.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: set fsid to 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:08:05.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T03:08:05.263 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:08:05.263 INFO:teuthology.orchestra.run.smithi094.stdout:Creating mon... 2024-03-12T03:08:06.251 INFO:teuthology.orchestra.run.smithi094.stdout:create mon.a on 2024-03-12T03:08:06.447 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-12T03:08:06.614 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-12T03:08:06.785 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-964142e8-e01d-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-964142e8-e01d-11ee-95c8-87774f69a715.target. 2024-03-12T03:08:06.785 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-964142e8-e01d-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-964142e8-e01d-11ee-95c8-87774f69a715.target. 2024-03-12T03:08:07.034 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.a 2024-03-12T03:08:07.035 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Failed to reset failed state of unit ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-12T03:08:07.189 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-964142e8-e01d-11ee-95c8-87774f69a715.target.wants/ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-964142e8-e01d-11ee-95c8-87774f69a715@.service. 2024-03-12T03:08:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:07 smithi094 podman[28899]: 2024-03-12 03:08:07.331588567 +0000 UTC m=+0.017521871 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:08:07.908 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:08:07.909 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout disabled 2024-03-12T03:08:07.916 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:08:07.916 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-03-12T03:08:07.917 INFO:teuthology.orchestra.run.smithi094.stdout:firewalld.service is not enabled 2024-03-12T03:08:07.917 INFO:teuthology.orchestra.run.smithi094.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T03:08:07.917 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mon to start... 2024-03-12T03:08:07.917 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mon... 2024-03-12T03:08:08.040 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:07 smithi094 podman[28899]: 2024-03-12 03:08:07.826723854 +0000 UTC m=+0.512657149 container init 407bcc7c9f471958629f84d19ad3534442d066808f0f5f19c4945c5c5e63e316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-12T03:08:08.040 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:07 smithi094 podman[28899]: 2024-03-12 03:08:07.829841342 +0000 UTC m=+0.515774635 container start 407bcc7c9f471958629f84d19ad3534442d066808f0f5f19c4945c5c5e63e316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-12T03:08:08.040 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:07 smithi094 bash[28899]: 407bcc7c9f471958629f84d19ad3534442d066808f0f5f19c4945c5c5e63e316 2024-03-12T03:08:08.041 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:07 smithi094 ceph-mon[28919]: mkfs 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:08:08.041 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:07 smithi094 systemd[1]: Started Ceph mon.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:08:08.041 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:07 smithi094 ceph-mon[28919]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-12T03:08:09.110 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:08 smithi094 ceph-mon[28919]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-12T03:08:09.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:08 smithi094 ceph-mon[28919]: monmap e1: 1 mons at {a=v1:172.21.15.94:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:08:09.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:08 smithi094 ceph-mon[28919]: fsmap 2024-03-12T03:08:09.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:08 smithi094 ceph-mon[28919]: osdmap e1: 0 total, 0 up, 0 in 2024-03-12T03:08:09.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:08 smithi094 ceph-mon[28919]: mgrmap e1: no daemons active 2024-03-12T03:08:09.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:08 smithi094 ceph-mon[28919]: from='client.? v1:172.21.15.94:0/3332613729' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:08:09.112 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout cluster: 2024-03-12T03:08:09.118 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout id: 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:08:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-12T03:08:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-03-12T03:08:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout services: 2024-03-12T03:08:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.570398s) 2024-03-12T03:08:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-12T03:08:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-12T03:08:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-03-12T03:08:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout data: 2024-03-12T03:08:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-12T03:08:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-12T03:08:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-12T03:08:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout pgs: 2024-03-12T03:08:09.120 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-03-12T03:08:09.120 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.384+0000 7fe71b3fe700 1 Processor -- start 2024-03-12T03:08:09.120 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.384+0000 7fe71b3fe700 1 -- start start 2024-03-12T03:08:09.120 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.384+0000 7fe71b3fe700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe714079690 con 0x7fe7141067f0 2024-03-12T03:08:09.120 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.385+0000 7fe71919a700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe7141067f0 0x7fe714108be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59946/0 (socket says 172.21.15.94:59946) 2024-03-12T03:08:09.120 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.385+0000 7fe71919a700 1 -- 172.21.15.94:0/3754184614 learned_addr learned my addr 172.21.15.94:0/3754184614 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:09.120 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.385+0000 7fe70bfff700 1 -- 172.21.15.94:0/3754184614 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173086182 0 0) 0x7fe714079690 con 0x7fe7141067f0 2024-03-12T03:08:09.120 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.385+0000 7fe70bfff700 1 -- 172.21.15.94:0/3754184614 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7000035b0 con 0x7fe7141067f0 2024-03-12T03:08:09.120 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.385+0000 7fe70bfff700 1 -- 172.21.15.94:0/3754184614 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1922148123 0 0) 0x7fe7000035b0 con 0x7fe7141067f0 2024-03-12T03:08:09.120 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.385+0000 7fe70bfff700 1 -- 172.21.15.94:0/3754184614 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71410b310 con 0x7fe7141067f0 2024-03-12T03:08:09.120 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.385+0000 7fe70bfff700 1 -- 172.21.15.94:0/3754184614 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fe704001580 con 0x7fe7141067f0 2024-03-12T03:08:09.120 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.385+0000 7fe70bfff700 1 -- 172.21.15.94:0/3754184614 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe704001920 con 0x7fe7141067f0 2024-03-12T03:08:09.120 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.385+0000 7fe70bfff700 1 -- 172.21.15.94:0/3754184614 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fe704001ca0 con 0x7fe7141067f0 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.386+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3754184614 >> v1:172.21.15.94:6789/0 conn(0x7fe7141067f0 legacy=0x7fe714108be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.386+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3754184614 shutdown_connections 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.386+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3754184614 >> 172.21.15.94:0/3754184614 conn(0x7fe714100200 msgr2=0x7fe714102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.386+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3754184614 shutdown_connections 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.386+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3754184614 wait complete. 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.387+0000 7fe71b3fe700 1 Processor -- start 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.387+0000 7fe71b3fe700 1 -- start start 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.387+0000 7fe71b3fe700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe7141996e0 con 0x7fe7141067f0 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.387+0000 7fe71919a700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe7141067f0 0x7fe714198fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59958/0 (socket says 172.21.15.94:59958) 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.387+0000 7fe71919a700 1 -- 172.21.15.94:0/3332613729 learned_addr learned my addr 172.21.15.94:0/3332613729 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.387+0000 7fe70a7fc700 1 -- 172.21.15.94:0/3332613729 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888801718 0 0) 0x7fe7141996e0 con 0x7fe7141067f0 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.387+0000 7fe70a7fc700 1 -- 172.21.15.94:0/3332613729 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7000031f0 con 0x7fe7141067f0 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.387+0000 7fe70a7fc700 1 -- 172.21.15.94:0/3332613729 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3180096050 0 0) 0x7fe7000031f0 con 0x7fe7141067f0 2024-03-12T03:08:09.121 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.387+0000 7fe70a7fc700 1 -- 172.21.15.94:0/3332613729 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7141996e0 con 0x7fe7141067f0 2024-03-12T03:08:09.122 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.387+0000 7fe70a7fc700 1 -- 172.21.15.94:0/3332613729 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fe704001540 con 0x7fe7141067f0 2024-03-12T03:08:09.122 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.388+0000 7fe70a7fc700 1 -- 172.21.15.94:0/3332613729 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 581051082 0 0) 0x7fe7141996e0 con 0x7fe7141067f0 2024-03-12T03:08:09.122 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.388+0000 7fe70a7fc700 1 -- 172.21.15.94:0/3332613729 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7141998b0 con 0x7fe7141067f0 2024-03-12T03:08:09.122 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.388+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3332613729 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe714199ba0 con 0x7fe7141067f0 2024-03-12T03:08:09.122 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.388+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3332613729 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71419d6f0 con 0x7fe7141067f0 2024-03-12T03:08:09.122 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.388+0000 7fe70a7fc700 1 -- 172.21.15.94:0/3332613729 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe704001920 con 0x7fe7141067f0 2024-03-12T03:08:09.122 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.388+0000 7fe70a7fc700 1 -- 172.21.15.94:0/3332613729 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fe704004cc0 con 0x7fe7141067f0 2024-03-12T03:08:09.122 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.388+0000 7fe70a7fc700 1 -- 172.21.15.94:0/3332613729 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe704005270 con 0x7fe7141067f0 2024-03-12T03:08:09.122 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.388+0000 7fe70a7fc700 1 -- 172.21.15.94:0/3332613729 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2600795447 0 0) 0x7fe704006960 con 0x7fe7141067f0 2024-03-12T03:08:09.122 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.388+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3332613729 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f8005150 con 0x7fe7141067f0 2024-03-12T03:08:09.122 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.391+0000 7fe70a7fc700 1 -- 172.21.15.94:0/3332613729 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe7040057f0 con 0x7fe7141067f0 2024-03-12T03:08:09.122 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.460+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3332613729 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe6f8005c50 con 0x7fe7141067f0 2024-03-12T03:08:09.122 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.460+0000 7fe70a7fc700 1 -- 172.21.15.94:0/3332613729 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (unknown 1155462804 0 283052844) 0x7fe704005a30 con 0x7fe7141067f0 2024-03-12T03:08:09.123 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.461+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3332613729 >> v1:172.21.15.94:6789/0 conn(0x7fe7141067f0 legacy=0x7fe714198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:09.123 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.461+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3332613729 shutdown_connections 2024-03-12T03:08:09.123 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.461+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3332613729 >> 172.21.15.94:0/3332613729 conn(0x7fe714100200 msgr2=0x7fe714190250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:09.123 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.461+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3332613729 shutdown_connections 2024-03-12T03:08:09.123 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:08.461+0000 7fe71b3fe700 1 -- 172.21.15.94:0/3332613729 wait complete. 2024-03-12T03:08:09.123 INFO:teuthology.orchestra.run.smithi094.stdout:mon is available 2024-03-12T03:08:09.123 INFO:teuthology.orchestra.run.smithi094.stdout:Assimilating anything we can from ceph.conf... 2024-03-12T03:08:10.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:09 smithi094 ceph-mon[28919]: from='client.? v1:172.21.15.94:0/4079150063' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-12T03:08:10.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:09 smithi094 ceph-mon[28919]: from='client.? v1:172.21.15.94:0/4079150063' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-12T03:08:10.275 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-03-12T03:08:10.275 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T03:08:10.275 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout fsid = 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.94:6789] 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.540+0000 7f4ca520a700 1 Processor -- start 2024-03-12T03:08:10.276 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.540+0000 7f4ca520a700 1 -- start start 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.540+0000 7f4ca520a700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4ca0079670 con 0x7f4ca01067e0 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.540+0000 7f4c9ed9d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4ca01067e0 0x7f4ca0108bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59974/0 (socket says 172.21.15.94:59974) 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.540+0000 7f4c9ed9d700 1 -- 172.21.15.94:0/2302527943 learned_addr learned my addr 172.21.15.94:0/2302527943 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.540+0000 7f4c9dd9b700 1 -- 172.21.15.94:0/2302527943 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776034398 0 0) 0x7f4ca0079670 con 0x7f4ca01067e0 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.540+0000 7f4c9dd9b700 1 -- 172.21.15.94:0/2302527943 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c880035b0 con 0x7f4ca01067e0 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.540+0000 7f4c9dd9b700 1 -- 172.21.15.94:0/2302527943 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1349110480 0 0) 0x7f4c880035b0 con 0x7f4ca01067e0 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.540+0000 7f4c9dd9b700 1 -- 172.21.15.94:0/2302527943 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca010b300 con 0x7f4ca01067e0 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.541+0000 7f4c9dd9b700 1 -- 172.21.15.94:0/2302527943 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f4c90001580 con 0x7f4ca01067e0 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.541+0000 7f4c9dd9b700 1 -- 172.21.15.94:0/2302527943 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c90001920 con 0x7f4ca01067e0 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.541+0000 7f4c9dd9b700 1 -- 172.21.15.94:0/2302527943 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f4c90001ca0 con 0x7f4ca01067e0 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.541+0000 7f4ca520a700 1 -- 172.21.15.94:0/2302527943 >> v1:172.21.15.94:6789/0 conn(0x7f4ca01067e0 legacy=0x7f4ca0108bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.541+0000 7f4ca520a700 1 -- 172.21.15.94:0/2302527943 shutdown_connections 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.541+0000 7f4ca520a700 1 -- 172.21.15.94:0/2302527943 >> 172.21.15.94:0/2302527943 conn(0x7f4ca01001b0 msgr2=0x7f4ca01025f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.541+0000 7f4ca520a700 1 -- 172.21.15.94:0/2302527943 shutdown_connections 2024-03-12T03:08:10.277 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.541+0000 7f4ca520a700 1 -- 172.21.15.94:0/2302527943 wait complete. 2024-03-12T03:08:10.278 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.542+0000 7f4ca520a700 1 Processor -- start 2024-03-12T03:08:10.278 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.542+0000 7f4ca520a700 1 -- start start 2024-03-12T03:08:10.278 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.542+0000 7f4ca520a700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4ca0199730 con 0x7f4ca01067e0 2024-03-12T03:08:10.278 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.542+0000 7f4c9ed9d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4ca01067e0 0x7f4ca0199020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59978/0 (socket says 172.21.15.94:59978) 2024-03-12T03:08:10.278 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.542+0000 7f4c9ed9d700 1 -- 172.21.15.94:0/4079150063 learned_addr learned my addr 172.21.15.94:0/4079150063 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:10.278 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.543+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366769888 0 0) 0x7f4ca0199730 con 0x7f4ca01067e0 2024-03-12T03:08:10.278 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.543+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c880031f0 con 0x7f4ca01067e0 2024-03-12T03:08:10.278 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.543+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3123255245 0 0) 0x7f4c880031f0 con 0x7f4ca01067e0 2024-03-12T03:08:10.278 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.543+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ca0199730 con 0x7f4ca01067e0 2024-03-12T03:08:10.278 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.543+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f4c90001540 con 0x7f4ca01067e0 2024-03-12T03:08:10.278 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.543+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2922486502 0 0) 0x7f4ca0199730 con 0x7f4ca01067e0 2024-03-12T03:08:10.278 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.543+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca0199900 con 0x7f4ca01067e0 2024-03-12T03:08:10.278 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.543+0000 7f4ca520a700 1 -- 172.21.15.94:0/4079150063 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca0199bf0 con 0x7f4ca01067e0 2024-03-12T03:08:10.279 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.543+0000 7f4ca520a700 1 -- 172.21.15.94:0/4079150063 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca019d740 con 0x7f4ca01067e0 2024-03-12T03:08:10.279 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.543+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c90004d40 con 0x7f4ca01067e0 2024-03-12T03:08:10.279 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.543+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f4c90001920 con 0x7f4ca01067e0 2024-03-12T03:08:10.279 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.543+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4c90006370 con 0x7f4ca01067e0 2024-03-12T03:08:10.279 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.543+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2600795447 0 0) 0x7f4c90006b80 con 0x7f4ca01067e0 2024-03-12T03:08:10.279 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.544+0000 7f4ca520a700 1 -- 172.21.15.94:0/4079150063 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c8c005150 con 0x7f4ca01067e0 2024-03-12T03:08:10.279 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.546+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4c900067f0 con 0x7f4ca01067e0 2024-03-12T03:08:10.279 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.612+0000 7f4ca520a700 1 -- 172.21.15.94:0/4079150063 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f4c8c002430 con 0x7f4ca01067e0 2024-03-12T03:08:10.279 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.614+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+317 (unknown 1213389831 0 268460625) 0x7f4c90004f70 con 0x7f4ca01067e0 2024-03-12T03:08:10.279 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.615+0000 7f4c87fff700 1 -- 172.21.15.94:0/4079150063 <== mon.0 v1:172.21.15.94:6789/0 11 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f4c900055c0 con 0x7f4ca01067e0 2024-03-12T03:08:10.279 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.615+0000 7f4ca520a700 1 -- 172.21.15.94:0/4079150063 >> v1:172.21.15.94:6789/0 conn(0x7f4ca01067e0 legacy=0x7f4ca0199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:10.279 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.615+0000 7f4ca520a700 1 -- 172.21.15.94:0/4079150063 shutdown_connections 2024-03-12T03:08:10.280 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.615+0000 7f4ca520a700 1 -- 172.21.15.94:0/4079150063 >> 172.21.15.94:0/4079150063 conn(0x7f4ca01001b0 msgr2=0x7f4ca0190240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:10.280 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.615+0000 7f4ca520a700 1 -- 172.21.15.94:0/4079150063 shutdown_connections 2024-03-12T03:08:10.280 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:09.615+0000 7f4ca520a700 1 -- 172.21.15.94:0/4079150063 wait complete. 2024-03-12T03:08:10.280 INFO:teuthology.orchestra.run.smithi094.stdout:Generating new minimal ceph.conf... 2024-03-12T03:08:11.056 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:10 smithi094 ceph-mon[28919]: from='client.? v1:172.21.15.94:0/796043838' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:08:11.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.671+0000 7f9c6d258700 1 Processor -- start 2024-03-12T03:08:11.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.672+0000 7f9c6d258700 1 -- start start 2024-03-12T03:08:11.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.672+0000 7f9c6d258700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9c6810a2d0 con 0x7f9c681067c0 2024-03-12T03:08:11.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.672+0000 7f9c66d9d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9c681067c0 0x7f9c68108bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59994/0 (socket says 172.21.15.94:59994) 2024-03-12T03:08:11.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.672+0000 7f9c66d9d700 1 -- 172.21.15.94:0/2675479034 learned_addr learned my addr 172.21.15.94:0/2675479034 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:11.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.672+0000 7f9c65d9b700 1 -- 172.21.15.94:0/2675479034 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986415680 0 0) 0x7f9c6810a2d0 con 0x7f9c681067c0 2024-03-12T03:08:11.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.672+0000 7f9c65d9b700 1 -- 172.21.15.94:0/2675479034 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c500035b0 con 0x7f9c681067c0 2024-03-12T03:08:11.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.672+0000 7f9c65d9b700 1 -- 172.21.15.94:0/2675479034 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1572149097 0 0) 0x7f9c500035b0 con 0x7f9c681067c0 2024-03-12T03:08:11.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.672+0000 7f9c65d9b700 1 -- 172.21.15.94:0/2675479034 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6810b4b0 con 0x7f9c681067c0 2024-03-12T03:08:11.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.672+0000 7f9c65d9b700 1 -- 172.21.15.94:0/2675479034 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f9c58001580 con 0x7f9c681067c0 2024-03-12T03:08:11.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.672+0000 7f9c65d9b700 1 -- 172.21.15.94:0/2675479034 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f9c58001d20 con 0x7f9c681067c0 2024-03-12T03:08:11.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.672+0000 7f9c65d9b700 1 -- 172.21.15.94:0/2675479034 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f9c58005960 con 0x7f9c681067c0 2024-03-12T03:08:11.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.673+0000 7f9c6d258700 1 -- 172.21.15.94:0/2675479034 >> v1:172.21.15.94:6789/0 conn(0x7f9c681067c0 legacy=0x7f9c68108bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:11.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.673+0000 7f9c6d258700 1 -- 172.21.15.94:0/2675479034 shutdown_connections 2024-03-12T03:08:11.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.673+0000 7f9c6d258700 1 -- 172.21.15.94:0/2675479034 >> 172.21.15.94:0/2675479034 conn(0x7f9c681001b0 msgr2=0x7f9c681025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.673+0000 7f9c6d258700 1 -- 172.21.15.94:0/2675479034 shutdown_connections 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.673+0000 7f9c6d258700 1 -- 172.21.15.94:0/2675479034 wait complete. 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.674+0000 7f9c6d258700 1 Processor -- start 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.674+0000 7f9c6d258700 1 -- start start 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.674+0000 7f9c6d258700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9c68199fd0 con 0x7f9c681067c0 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.674+0000 7f9c66d9d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9c681067c0 0x7f9c681998c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59996/0 (socket says 172.21.15.94:59996) 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.674+0000 7f9c66d9d700 1 -- 172.21.15.94:0/796043838 learned_addr learned my addr 172.21.15.94:0/796043838 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.674+0000 7f9c4ffff700 1 -- 172.21.15.94:0/796043838 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771640242 0 0) 0x7f9c68199fd0 con 0x7f9c681067c0 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.674+0000 7f9c4ffff700 1 -- 172.21.15.94:0/796043838 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c50005530 con 0x7f9c681067c0 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.675+0000 7f9c4ffff700 1 -- 172.21.15.94:0/796043838 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3055380952 0 0) 0x7f9c50005530 con 0x7f9c681067c0 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.675+0000 7f9c4ffff700 1 -- 172.21.15.94:0/796043838 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c68199fd0 con 0x7f9c681067c0 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.675+0000 7f9c4ffff700 1 -- 172.21.15.94:0/796043838 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f9c58004870 con 0x7f9c681067c0 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.675+0000 7f9c4ffff700 1 -- 172.21.15.94:0/796043838 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3601934357 0 0) 0x7f9c68199fd0 con 0x7f9c681067c0 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.675+0000 7f9c4ffff700 1 -- 172.21.15.94:0/796043838 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6819a1a0 con 0x7f9c681067c0 2024-03-12T03:08:11.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.675+0000 7f9c6d258700 1 -- 172.21.15.94:0/796043838 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6819a490 con 0x7f9c681067c0 2024-03-12T03:08:11.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.675+0000 7f9c6d258700 1 -- 172.21.15.94:0/796043838 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6819dfe0 con 0x7f9c681067c0 2024-03-12T03:08:11.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.675+0000 7f9c4ffff700 1 -- 172.21.15.94:0/796043838 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f9c58004ab0 con 0x7f9c681067c0 2024-03-12T03:08:11.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.675+0000 7f9c4ffff700 1 -- 172.21.15.94:0/796043838 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f9c58006390 con 0x7f9c681067c0 2024-03-12T03:08:11.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.675+0000 7f9c4ffff700 1 -- 172.21.15.94:0/796043838 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9c580068c0 con 0x7f9c681067c0 2024-03-12T03:08:11.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.675+0000 7f9c4ffff700 1 -- 172.21.15.94:0/796043838 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2600795447 0 0) 0x7f9c58007db0 con 0x7f9c681067c0 2024-03-12T03:08:11.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.676+0000 7f9c6d258700 1 -- 172.21.15.94:0/796043838 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c54005150 con 0x7f9c681067c0 2024-03-12T03:08:11.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.678+0000 7f9c4ffff700 1 -- 172.21.15.94:0/796043838 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9c580056d0 con 0x7f9c681067c0 2024-03-12T03:08:11.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.745+0000 7f9c6d258700 1 -- 172.21.15.94:0/796043838 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9c54005440 con 0x7f9c681067c0 2024-03-12T03:08:11.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.746+0000 7f9c4ffff700 1 -- 172.21.15.94:0/796043838 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+147 (unknown 1452402520 0 945513535) 0x7f9c58005950 con 0x7f9c681067c0 2024-03-12T03:08:11.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.746+0000 7f9c6d258700 1 -- 172.21.15.94:0/796043838 >> v1:172.21.15.94:6789/0 conn(0x7f9c681067c0 legacy=0x7f9c681998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:11.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.747+0000 7f9c6d258700 1 -- 172.21.15.94:0/796043838 shutdown_connections 2024-03-12T03:08:11.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.747+0000 7f9c6d258700 1 -- 172.21.15.94:0/796043838 >> 172.21.15.94:0/796043838 conn(0x7f9c681001b0 msgr2=0x7f9c68190580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:11.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.747+0000 7f9c6d258700 1 -- 172.21.15.94:0/796043838 shutdown_connections 2024-03-12T03:08:11.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:10.747+0000 7f9c6d258700 1 -- 172.21.15.94:0/796043838 wait complete. 2024-03-12T03:08:11.335 INFO:teuthology.orchestra.run.smithi094.stdout:Restarting the monitor... 2024-03-12T03:08:11.585 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:11 smithi094 systemd[1]: Stopping Ceph mon.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:08:11.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:11 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a[28914]: 2024-03-12T03:08:11.581+0000 7f03443f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:08:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:11 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a[28914]: 2024-03-12T03:08:11.581+0000 7f03443f5700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-12T03:08:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:11 smithi094 podman[29163]: 2024-03-12 03:08:11.715675984 +0000 UTC m=+0.230961584 container died 407bcc7c9f471958629f84d19ad3534442d066808f0f5f19c4945c5c5e63e316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e) 2024-03-12T03:08:12.226 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 podman[29163]: 2024-03-12 03:08:12.040246716 +0000 UTC m=+0.555532314 container cleanup 407bcc7c9f471958629f84d19ad3534442d066808f0f5f19c4945c5c5e63e316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-12T03:08:12.226 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 bash[29163]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a 2024-03-12T03:08:12.673 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 podman[29177]: 2024-03-12 03:08:12.224446653 +0000 UTC m=+0.511911388 container remove 407bcc7c9f471958629f84d19ad3534442d066808f0f5f19c4945c5c5e63e316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-12T03:08:12.673 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-12T03:08:12.673 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 systemd[1]: Stopped Ceph mon.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:08:12.673 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 systemd[1]: Starting Ceph mon.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:08:12.923 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 podman[29272]: 2024-03-12 03:08:12.574993193 +0000 UTC m=+0.018870723 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 podman[29272]: 2024-03-12 03:08:12.675246617 +0000 UTC m=+0.119124143 container create 57104d25fd78692e58cef1a988ee5927b17b28071a2e5040bcb367321446dfb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 podman[29272]: 2024-03-12 03:08:12.835712479 +0000 UTC m=+0.279590004 container init 57104d25fd78692e58cef1a988ee5927b17b28071a2e5040bcb367321446dfb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 podman[29272]: 2024-03-12 03:08:12.838544382 +0000 UTC m=+0.282421909 container start 57104d25fd78692e58cef1a988ee5927b17b28071a2e5040bcb367321446dfb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 2 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: pidfile_write: ignore empty --pid-file 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: load: jerasure load: lrc 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: RocksDB version: 7.9.2 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Git sha 0 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: DB SUMMARY 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: DB Session ID: UIFEVOB1U1TKFRZYNPDE 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:08:12.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:08:12.925 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-12T03:08:12.925 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-12T03:08:12.925 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 85546 ; 2024-03-12T03:08:12.925 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:08:12.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:08:12.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:08:12.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T03:08:12.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:08:12.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T03:08:12.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.env: 0x560910f96b20 2024-03-12T03:08:12.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T03:08:12.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.info_log: 0x560912321920 2024-03-12T03:08:12.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:08:12.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.statistics: (nil) 2024-03-12T03:08:12.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.use_fsync: 0 2024-03-12T03:08:12.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.db_log_dir: 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.wal_dir: 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:08:12.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:08:12.928 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:08:12.928 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:08:12.928 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:08:12.928 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:08:12.928 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:08:12.928 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:08:12.928 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.write_buffer_manager: 0x5609122f5400 2024-03-12T03:08:12.928 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:08:12.928 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:08:12.928 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:08:12.928 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:08:12.928 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.unordered_write: 0 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.row_cache: None 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.wal_filter: None 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:08:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:08:12.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.wal_compression: 0 2024-03-12T03:08:12.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:08:12.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:08:12.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:08:12.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:08:12.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:08:12.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:08:12.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:08:12.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:08:12.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:08:12.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:08:12.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:08:12.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_open_files: -1 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:08:12.931 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Compression algorithms supported: 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: kZSTD supported: 0 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T03:08:12.932 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.merge_operator: 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compaction_filter: None 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560912321a20) 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:08:12.933 INFO:journalctl@ceph.mon.a.smithi094.stdout: index_type: 0 2024-03-12T03:08:12.934 INFO:journalctl@ceph.mon.a.smithi094.stdout: data_block_index_type: 0 2024-03-12T03:08:12.934 INFO:journalctl@ceph.mon.a.smithi094.stdout: index_shortening: 1 2024-03-12T03:08:12.934 INFO:journalctl@ceph.mon.a.smithi094.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:08:12.934 INFO:journalctl@ceph.mon.a.smithi094.stdout: checksum: 4 2024-03-12T03:08:12.934 INFO:journalctl@ceph.mon.a.smithi094.stdout: no_block_cache: 0 2024-03-12T03:08:12.934 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_cache: 0x560912327090 2024-03-12T03:08:12.934 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:08:12.934 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_cache_options: 2024-03-12T03:08:12.934 INFO:journalctl@ceph.mon.a.smithi094.stdout: capacity : 536870912 2024-03-12T03:08:12.934 INFO:journalctl@ceph.mon.a.smithi094.stdout: num_shard_bits : 4 2024-03-12T03:08:12.934 INFO:journalctl@ceph.mon.a.smithi094.stdout: strict_capacity_limit : 0 2024-03-12T03:08:12.934 INFO:journalctl@ceph.mon.a.smithi094.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_cache_compressed: (nil) 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: persistent_cache: (nil) 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_size: 4096 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_size_deviation: 10 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_restart_interval: 16 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: index_block_restart_interval: 1 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: metadata_block_size: 4096 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: partition_filters: 0 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: use_delta_encoding: 1 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: filter_policy: bloomfilter 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: whole_key_filtering: 1 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: verify_compression: 0 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: format_version: 5 2024-03-12T03:08:12.935 INFO:journalctl@ceph.mon.a.smithi094.stdout: enable_index_compression: 1 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_align: 0 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout: max_auto_readahead_size: 262144 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout: prepopulate_block_cache: 0 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout: initial_auto_readahead_size: 8192 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compression: NoCompression 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.num_levels: 7 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:08:12.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:08:12.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:08:12.938 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:08:12.938 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:08:12.938 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:08:12.938 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:08:12.938 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:08:12.938 INFO:teuthology.orchestra.run.smithi094.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-12T03:08:12.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:08:12.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:08:12.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:08:12.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:08:12.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:08:12.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:08:12.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:08:12.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:08:12.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:08:12.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:08:12.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:08:12.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:08:12.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:08:12.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:08:12.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:08:12.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.ttl: 2592000 2024-03-12T03:08:12.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:08:12.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T03:08:12.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T03:08:12.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:08:12.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:08:12.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:08:12.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:08:12.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:08:12.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:08:12.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T03:08:12.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T03:08:12.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T03:08:12.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T03:08:12.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-12T03:08:12.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-12T03:08:12.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 30174515-6662-4bf8-aef3-427ac042dd3d 2024-03-12T03:08:12.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710212892867365, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-12T03:08:12.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-12T03:08:12.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710212892868207, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 264, "table_properties": {"data_size": 80079, "index_size": 211, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10703, "raw_average_key_size": 47, "raw_value_size": 74054, "raw_average_value_size": 332, "num_data_blocks": 10, "num_entries": 223, "num_filter_entries": 223, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710212892, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "30174515-6662-4bf8-aef3-427ac042dd3d", "db_session_id": "UIFEVOB1U1TKFRZYNPDE", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-12T03:08:12.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710212892868291, "job": 1, "event": "recovery_finished"} 2024-03-12T03:08:12.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-12T03:08:12.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56091241a000 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: DB pointer 0x560912404000 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout: ** DB Stats ** 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-03-12T03:08:12.944 INFO:journalctl@ceph.mon.a.smithi094.stdout: ** Compaction Stats [default] ** 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: L0 2/0 81.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: Sum 2/0 81.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: ** Compaction Stats [default] ** 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 145.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:08:12.945 INFO:journalctl@ceph.mon.a.smithi094.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:08:12.946 INFO:journalctl@ceph.mon.a.smithi094.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:08:12.946 INFO:journalctl@ceph.mon.a.smithi094.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:08:12.946 INFO:journalctl@ceph.mon.a.smithi094.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:08:12.946 INFO:journalctl@ceph.mon.a.smithi094.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:08:12.946 INFO:journalctl@ceph.mon.a.smithi094.stdout: Cumulative compaction: 0.00 GB write, 27.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:08:12.946 INFO:journalctl@ceph.mon.a.smithi094.stdout: Interval compaction: 0.00 GB write, 27.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:08:12.946 INFO:journalctl@ceph.mon.a.smithi094.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:08:12.946 INFO:journalctl@ceph.mon.a.smithi094.stdout: Block cache BinnedLRUCache@0x560912327090#2 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-12T03:08:12.946 INFO:journalctl@ceph.mon.a.smithi094.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-12T03:08:12.946 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-03-12T03:08:12.946 INFO:journalctl@ceph.mon.a.smithi094.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:08:12.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: starting mon.a rank 0 at public addrs v1:172.21.15.94:6789/0 at bind addrs v1:172.21.15.94:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:08:12.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: mon.a@-1(???) e1 preinit fsid 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: mon.a@-1(???).mds e1 new map 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: mon.a@-1(???).mds e1 print_map 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout: e1 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout: legacy client fscid: -1 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout: No filesystems configured 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 bash[29272]: 57104d25fd78692e58cef1a988ee5927b17b28071a2e5040bcb367321446dfb9 2024-03-12T03:08:12.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 systemd[1]: Started Ceph mon.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:08:13.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-12T03:08:13.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: monmap e1: 1 mons at {a=v1:172.21.15.94:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:08:13.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: fsmap 2024-03-12T03:08:13.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: osdmap e1: 0 total, 0 up, 0 in 2024-03-12T03:08:13.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:12 smithi094 ceph-mon[29291]: mgrmap e1: no daemons active 2024-03-12T03:08:14.317 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.707+0000 7fba1f72b700 1 Processor -- start 2024-03-12T03:08:14.317 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.707+0000 7fba1f72b700 1 -- start start 2024-03-12T03:08:14.317 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.707+0000 7fba1f72b700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba181090c0 con 0x7fba181062e0 2024-03-12T03:08:14.317 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.707+0000 7fba1d4c7700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fba181062e0 0x7fba18106700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:60006/0 (socket says 172.21.15.94:60006) 2024-03-12T03:08:14.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.707+0000 7fba1d4c7700 1 -- 172.21.15.94:0/1777700786 learned_addr learned my addr 172.21.15.94:0/1777700786 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:14.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.708+0000 7fba0ffff700 1 -- 172.21.15.94:0/1777700786 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184650902 0 0) 0x7fba181090c0 con 0x7fba181062e0 2024-03-12T03:08:14.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.708+0000 7fba0ffff700 1 -- 172.21.15.94:0/1777700786 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba040035b0 con 0x7fba181062e0 2024-03-12T03:08:14.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.708+0000 7fba0ffff700 1 -- 172.21.15.94:0/1777700786 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2372483472 0 0) 0x7fba040035b0 con 0x7fba181062e0 2024-03-12T03:08:14.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.708+0000 7fba0ffff700 1 -- 172.21.15.94:0/1777700786 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1810a2a0 con 0x7fba181062e0 2024-03-12T03:08:14.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.708+0000 7fba0ffff700 1 -- 172.21.15.94:0/1777700786 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fba14001580 con 0x7fba181062e0 2024-03-12T03:08:14.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.708+0000 7fba0ffff700 1 -- 172.21.15.94:0/1777700786 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7fba14001d20 con 0x7fba181062e0 2024-03-12T03:08:14.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.708+0000 7fba0ffff700 1 -- 172.21.15.94:0/1777700786 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fba14005960 con 0x7fba181062e0 2024-03-12T03:08:14.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.709+0000 7fba1f72b700 1 -- 172.21.15.94:0/1777700786 >> v1:172.21.15.94:6789/0 conn(0x7fba181062e0 legacy=0x7fba18106700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:14.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.709+0000 7fba1f72b700 1 -- 172.21.15.94:0/1777700786 shutdown_connections 2024-03-12T03:08:14.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.709+0000 7fba1f72b700 1 -- 172.21.15.94:0/1777700786 >> 172.21.15.94:0/1777700786 conn(0x7fba180ff900 msgr2=0x7fba18101d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:14.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.709+0000 7fba1f72b700 1 -- 172.21.15.94:0/1777700786 shutdown_connections 2024-03-12T03:08:14.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.709+0000 7fba1f72b700 1 -- 172.21.15.94:0/1777700786 wait complete. 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.709+0000 7fba1f72b700 1 Processor -- start 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.709+0000 7fba1f72b700 1 -- start start 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.709+0000 7fba1f72b700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba1819f950 con 0x7fba181062e0 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.710+0000 7fba1d4c7700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fba181062e0 0x7fba181017e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:60014/0 (socket says 172.21.15.94:60014) 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.710+0000 7fba1d4c7700 1 -- 172.21.15.94:0/3566289958 learned_addr learned my addr 172.21.15.94:0/3566289958 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.710+0000 7fba0e7fc700 1 -- 172.21.15.94:0/3566289958 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918380843 0 0) 0x7fba1819f950 con 0x7fba181062e0 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.710+0000 7fba0e7fc700 1 -- 172.21.15.94:0/3566289958 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba04005530 con 0x7fba181062e0 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.710+0000 7fba0e7fc700 1 -- 172.21.15.94:0/3566289958 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1272846613 0 0) 0x7fba04005530 con 0x7fba181062e0 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.710+0000 7fba0e7fc700 1 -- 172.21.15.94:0/3566289958 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba1819f950 con 0x7fba181062e0 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.710+0000 7fba0e7fc700 1 -- 172.21.15.94:0/3566289958 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fba14004870 con 0x7fba181062e0 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.710+0000 7fba0e7fc700 1 -- 172.21.15.94:0/3566289958 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2342387840 0 0) 0x7fba1819f950 con 0x7fba181062e0 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.710+0000 7fba0e7fc700 1 -- 172.21.15.94:0/3566289958 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba181a0b30 con 0x7fba181062e0 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.710+0000 7fba1f72b700 1 -- 172.21.15.94:0/3566289958 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1819fb80 con 0x7fba181062e0 2024-03-12T03:08:14.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.710+0000 7fba1f72b700 1 -- 172.21.15.94:0/3566289958 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba181a00c0 con 0x7fba181062e0 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.710+0000 7fba0e7fc700 1 -- 172.21.15.94:0/3566289958 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7fba14004ab0 con 0x7fba181062e0 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.710+0000 7fba0e7fc700 1 -- 172.21.15.94:0/3566289958 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fba14005fe0 con 0x7fba181062e0 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.711+0000 7fba0e7fc700 1 -- 172.21.15.94:0/3566289958 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fba140065d0 con 0x7fba181062e0 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.711+0000 7fba0e7fc700 1 -- 172.21.15.94:0/3566289958 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2600795447 0 0) 0x7fba14007d30 con 0x7fba181062e0 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.711+0000 7fba1f72b700 1 -- 172.21.15.94:0/3566289958 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9fc005150 con 0x7fba181062e0 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.714+0000 7fba0e7fc700 1 -- 172.21.15.94:0/3566289958 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fba140056d0 con 0x7fba181062e0 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.780+0000 7fba1f72b700 1 -- 172.21.15.94:0/3566289958 --> v1:172.21.15.94:6789/0 -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fb9fc005440 con 0x7fba181062e0 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.781+0000 7fba0e7fc700 1 -- 172.21.15.94:0/3566289958 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (unknown 534127980 0 0) 0x7fba14005950 con 0x7fba181062e0 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.782+0000 7fba1f72b700 1 -- 172.21.15.94:0/3566289958 >> v1:172.21.15.94:6789/0 conn(0x7fba181062e0 legacy=0x7fba181017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.782+0000 7fba1f72b700 1 -- 172.21.15.94:0/3566289958 shutdown_connections 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.782+0000 7fba1f72b700 1 -- 172.21.15.94:0/3566289958 >> 172.21.15.94:0/3566289958 conn(0x7fba180ff900 msgr2=0x7fba18103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.782+0000 7fba1f72b700 1 -- 172.21.15.94:0/3566289958 shutdown_connections 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:13.782+0000 7fba1f72b700 1 -- 172.21.15.94:0/3566289958 wait complete. 2024-03-12T03:08:14.320 INFO:teuthology.orchestra.run.smithi094.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-12T03:08:14.327 INFO:teuthology.orchestra.run.smithi094.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:08:14.327 INFO:teuthology.orchestra.run.smithi094.stdout:Creating mgr... 2024-03-12T03:08:14.327 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-12T03:08:14.327 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-12T03:08:14.552 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-964142e8-e01d-11ee-95c8-87774f69a715@mgr.y 2024-03-12T03:08:14.552 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Failed to reset failed state of unit ceph-964142e8-e01d-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-964142e8-e01d-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-12T03:08:14.694 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-964142e8-e01d-11ee-95c8-87774f69a715.target.wants/ceph-964142e8-e01d-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-964142e8-e01d-11ee-95c8-87774f69a715@.service. 2024-03-12T03:08:15.062 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3566289958' entity='client.admin' 2024-03-12T03:08:15.254 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:08:15.254 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout disabled 2024-03-12T03:08:15.261 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:08:15.261 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-03-12T03:08:15.261 INFO:teuthology.orchestra.run.smithi094.stdout:firewalld.service is not enabled 2024-03-12T03:08:15.261 INFO:teuthology.orchestra.run.smithi094.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T03:08:15.269 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:08:15.269 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout disabled 2024-03-12T03:08:15.276 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:08:15.276 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-03-12T03:08:15.276 INFO:teuthology.orchestra.run.smithi094.stdout:firewalld.service is not enabled 2024-03-12T03:08:15.277 INFO:teuthology.orchestra.run.smithi094.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-12T03:08:15.277 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mgr to start... 2024-03-12T03:08:15.277 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mgr... 2024-03-12T03:08:15.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:15 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2935507679' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:08:15.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:15 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:15.817+0000 7f9edc763200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:08:16.375 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-03-12T03:08:16.376 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:08:16.376 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsid": "964142e8-e01d-11ee-95c8-87774f69a715", 2024-03-12T03:08:16.376 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:08:16.376 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:08:16.376 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:08:16.376 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:08:16.376 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:16.376 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:08:16.376 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:08:16.376 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:08:16.376 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:08:16.376 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:08:16.376 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:08:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:08:16.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:08:16.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:16.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:08:16.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:08:16.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:08:16.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:08:16.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:08:16.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:08:16.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:08:16.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:08:16.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:08:16.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:08:16.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:08:16.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:16.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:08:16.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:16.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:08:07.892677+0000", 2024-03-12T03:08:16.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:08:16.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:16.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:08:16.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:08:16.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.700+0000 7f92bdc77700 1 Processor -- start 2024-03-12T03:08:16.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.700+0000 7f92bdc77700 1 -- start start 2024-03-12T03:08:16.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.700+0000 7f92bdc77700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f92b8079690 con 0x7f92b81067b0 2024-03-12T03:08:16.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.700+0000 7f92b77fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f92b81067b0 0x7f92b8108ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36850/0 (socket says 172.21.15.94:36850) 2024-03-12T03:08:16.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.701+0000 7f92b77fe700 1 -- 172.21.15.94:0/2091681225 learned_addr learned my addr 172.21.15.94:0/2091681225 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:16.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.701+0000 7f92b67fc700 1 -- 172.21.15.94:0/2091681225 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811385690 0 0) 0x7f92b8079690 con 0x7f92b81067b0 2024-03-12T03:08:16.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.701+0000 7f92b67fc700 1 -- 172.21.15.94:0/2091681225 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a00035b0 con 0x7f92b81067b0 2024-03-12T03:08:16.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.701+0000 7f92b67fc700 1 -- 172.21.15.94:0/2091681225 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3078806783 0 0) 0x7f92a00035b0 con 0x7f92b81067b0 2024-03-12T03:08:16.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.701+0000 7f92b67fc700 1 -- 172.21.15.94:0/2091681225 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b810b2d0 con 0x7f92b81067b0 2024-03-12T03:08:16.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.701+0000 7f92b67fc700 1 -- 172.21.15.94:0/2091681225 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f92ac001580 con 0x7f92b81067b0 2024-03-12T03:08:16.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.701+0000 7f92b67fc700 1 -- 172.21.15.94:0/2091681225 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f92ac001d20 con 0x7f92b81067b0 2024-03-12T03:08:16.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.701+0000 7f92b67fc700 1 -- 172.21.15.94:0/2091681225 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f92ac005960 con 0x7f92b81067b0 2024-03-12T03:08:16.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.702+0000 7f92bdc77700 1 -- 172.21.15.94:0/2091681225 >> v1:172.21.15.94:6789/0 conn(0x7f92b81067b0 legacy=0x7f92b8108ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:16.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.702+0000 7f92bdc77700 1 -- 172.21.15.94:0/2091681225 shutdown_connections 2024-03-12T03:08:16.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.702+0000 7f92bdc77700 1 -- 172.21.15.94:0/2091681225 >> 172.21.15.94:0/2091681225 conn(0x7f92b81001e0 msgr2=0x7f92b8102600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:16.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.702+0000 7f92bdc77700 1 -- 172.21.15.94:0/2091681225 shutdown_connections 2024-03-12T03:08:16.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.702+0000 7f92bdc77700 1 -- 172.21.15.94:0/2091681225 wait complete. 2024-03-12T03:08:16.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.702+0000 7f92bdc77700 1 Processor -- start 2024-03-12T03:08:16.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.703+0000 7f92bdc77700 1 -- start start 2024-03-12T03:08:16.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.703+0000 7f92bdc77700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f92b819f830 con 0x7f92b81067b0 2024-03-12T03:08:16.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.703+0000 7f92b77fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f92b81067b0 0x7f92b8106030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36860/0 (socket says 172.21.15.94:36860) 2024-03-12T03:08:16.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.703+0000 7f92b77fe700 1 -- 172.21.15.94:0/2935507679 learned_addr learned my addr 172.21.15.94:0/2935507679 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:16.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.703+0000 7f92b4ff9700 1 -- 172.21.15.94:0/2935507679 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125501924 0 0) 0x7f92b819f830 con 0x7f92b81067b0 2024-03-12T03:08:16.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.703+0000 7f92b4ff9700 1 -- 172.21.15.94:0/2935507679 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a0005530 con 0x7f92b81067b0 2024-03-12T03:08:16.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.703+0000 7f92b4ff9700 1 -- 172.21.15.94:0/2935507679 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3471064139 0 0) 0x7f92a0005530 con 0x7f92b81067b0 2024-03-12T03:08:16.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.703+0000 7f92b4ff9700 1 -- 172.21.15.94:0/2935507679 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92b819f830 con 0x7f92b81067b0 2024-03-12T03:08:16.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.703+0000 7f92b4ff9700 1 -- 172.21.15.94:0/2935507679 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f92ac004870 con 0x7f92b81067b0 2024-03-12T03:08:16.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.703+0000 7f92b4ff9700 1 -- 172.21.15.94:0/2935507679 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1152399770 0 0) 0x7f92b819f830 con 0x7f92b81067b0 2024-03-12T03:08:16.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.703+0000 7f92b4ff9700 1 -- 172.21.15.94:0/2935507679 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b81a0a10 con 0x7f92b81067b0 2024-03-12T03:08:16.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.704+0000 7f92bdc77700 1 -- 172.21.15.94:0/2935507679 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b819fa60 con 0x7f92b81067b0 2024-03-12T03:08:16.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.704+0000 7f92bdc77700 1 -- 172.21.15.94:0/2935507679 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b819ffa0 con 0x7f92b81067b0 2024-03-12T03:08:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.704+0000 7f92b4ff9700 1 -- 172.21.15.94:0/2935507679 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f92ac001ce0 con 0x7f92b81067b0 2024-03-12T03:08:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.704+0000 7f92b4ff9700 1 -- 172.21.15.94:0/2935507679 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f92ac006080 con 0x7f92b81067b0 2024-03-12T03:08:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.704+0000 7f92b4ff9700 1 -- 172.21.15.94:0/2935507679 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f92ac006670 con 0x7f92b81067b0 2024-03-12T03:08:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.704+0000 7f92b4ff9700 1 -- 172.21.15.94:0/2935507679 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2600795447 0 0) 0x7f92ac007d10 con 0x7f92b81067b0 2024-03-12T03:08:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.704+0000 7f92bdc77700 1 -- 172.21.15.94:0/2935507679 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a4005120 con 0x7f92b81067b0 2024-03-12T03:08:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.707+0000 7f92b4ff9700 1 -- 172.21.15.94:0/2935507679 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f92ac0056d0 con 0x7f92b81067b0 2024-03-12T03:08:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.779+0000 7f92bdc77700 1 -- 172.21.15.94:0/2935507679 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f92a4005410 con 0x7f92b81067b0 2024-03-12T03:08:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.779+0000 7f92b4ff9700 1 -- 172.21.15.94:0/2935507679 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (unknown 4201413639 0 294682191) 0x7f92ac0058b0 con 0x7f92b81067b0 2024-03-12T03:08:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.780+0000 7f92bdc77700 1 -- 172.21.15.94:0/2935507679 >> v1:172.21.15.94:6789/0 conn(0x7f92b81067b0 legacy=0x7f92b8106030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.780+0000 7f92bdc77700 1 -- 172.21.15.94:0/2935507679 shutdown_connections 2024-03-12T03:08:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.780+0000 7f92bdc77700 1 -- 172.21.15.94:0/2935507679 >> 172.21.15.94:0/2935507679 conn(0x7f92b81001e0 msgr2=0x7f92b8190460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.780+0000 7f92bdc77700 1 -- 172.21.15.94:0/2935507679 shutdown_connections 2024-03-12T03:08:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:15.780+0000 7f92bdc77700 1 -- 172.21.15.94:0/2935507679 wait complete. 2024-03-12T03:08:16.384 INFO:teuthology.orchestra.run.smithi094.stdout:mgr not available, waiting (1/15)... 2024-03-12T03:08:16.706 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:16.467+0000 7f9edc763200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:08:16.706 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:16.559+0000 7f9edc763200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:08:16.706 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:16.704+0000 7f9edc763200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:08:16.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:16.797+0000 7f9edc763200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:08:16.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:16.877+0000 7f9edc763200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:08:17.422 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:17 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:17.135+0000 7f9edc763200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:08:17.422 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:17 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:17.242+0000 7f9edc763200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:08:17.422 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:17 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:17.326+0000 7f9edc763200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:08:17.422 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:17 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:17.419+0000 7f9edc763200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:08:17.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:17 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:17.591+0000 7f9edc763200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:08:18.791 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:18 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:18.552+0000 7f9edc763200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:08:18.791 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:18 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:18.682+0000 7f9edc763200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:08:19.119 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/896413276' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:08:19.119 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:19 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:19.027+0000 7f9edc763200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:08:19.120 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:19 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:19.116+0000 7f9edc763200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:08:19.459 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:19 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:19.314+0000 7f9edc763200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:08:19.607 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-03-12T03:08:19.607 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:08:19.607 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsid": "964142e8-e01d-11ee-95c8-87774f69a715", 2024-03-12T03:08:19.607 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:08:19.607 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:08:19.607 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:08:19.608 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:19.609 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:08:19.609 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:19.609 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:08:19.609 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:08:19.609 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:08:19.616 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:19.617 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:08:19.618 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:08:07.892677+0000", 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.797+0000 7f7ec4626700 1 Processor -- start 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.798+0000 7f7ec4626700 1 -- start start 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.798+0000 7f7ec4626700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7ebc10a1f0 con 0x7f7ebc107410 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.798+0000 7f7ec23c2700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7ebc107410 0x7f7ebc107830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36870/0 (socket says 172.21.15.94:36870) 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.798+0000 7f7ec23c2700 1 -- 172.21.15.94:0/4139149308 learned_addr learned my addr 172.21.15.94:0/4139149308 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.798+0000 7f7ec13c0700 1 -- 172.21.15.94:0/4139149308 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810295950 0 0) 0x7f7ebc10a1f0 con 0x7f7ebc107410 2024-03-12T03:08:19.619 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.798+0000 7f7ec13c0700 1 -- 172.21.15.94:0/4139149308 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea80035b0 con 0x7f7ebc107410 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.799+0000 7f7ec13c0700 1 -- 172.21.15.94:0/4139149308 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3704593389 0 0) 0x7f7ea80035b0 con 0x7f7ebc107410 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.799+0000 7f7ec13c0700 1 -- 172.21.15.94:0/4139149308 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc10b3d0 con 0x7f7ebc107410 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.799+0000 7f7ec13c0700 1 -- 172.21.15.94:0/4139149308 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f7eb8001580 con 0x7f7ebc107410 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.799+0000 7f7ec13c0700 1 -- 172.21.15.94:0/4139149308 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f7eb8001d20 con 0x7f7ebc107410 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.799+0000 7f7ec13c0700 1 -- 172.21.15.94:0/4139149308 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f7eb8005960 con 0x7f7ebc107410 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.799+0000 7f7ec4626700 1 -- 172.21.15.94:0/4139149308 >> v1:172.21.15.94:6789/0 conn(0x7f7ebc107410 legacy=0x7f7ebc107830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.799+0000 7f7ec4626700 1 -- 172.21.15.94:0/4139149308 shutdown_connections 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.799+0000 7f7ec4626700 1 -- 172.21.15.94:0/4139149308 >> 172.21.15.94:0/4139149308 conn(0x7f7ebc101290 msgr2=0x7f7ebc103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.800+0000 7f7ec4626700 1 -- 172.21.15.94:0/4139149308 shutdown_connections 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.800+0000 7f7ec4626700 1 -- 172.21.15.94:0/4139149308 wait complete. 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.800+0000 7f7ec4626700 1 Processor -- start 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.800+0000 7f7ec4626700 1 -- start start 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.800+0000 7f7ec4626700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7ebc19a040 con 0x7f7ebc107410 2024-03-12T03:08:19.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.800+0000 7f7ec23c2700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7ebc107410 0x7f7ebc199930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36872/0 (socket says 172.21.15.94:36872) 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.800+0000 7f7ec23c2700 1 -- 172.21.15.94:0/896413276 learned_addr learned my addr 172.21.15.94:0/896413276 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.801+0000 7f7eb37fe700 1 -- 172.21.15.94:0/896413276 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845398561 0 0) 0x7f7ebc19a040 con 0x7f7ebc107410 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.801+0000 7f7eb37fe700 1 -- 172.21.15.94:0/896413276 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea8005530 con 0x7f7ebc107410 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.801+0000 7f7eb37fe700 1 -- 172.21.15.94:0/896413276 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 782809473 0 0) 0x7f7ea8005530 con 0x7f7ebc107410 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.801+0000 7f7eb37fe700 1 -- 172.21.15.94:0/896413276 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ebc19a040 con 0x7f7ebc107410 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.801+0000 7f7eb37fe700 1 -- 172.21.15.94:0/896413276 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f7eb8004870 con 0x7f7ebc107410 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.801+0000 7f7eb37fe700 1 -- 172.21.15.94:0/896413276 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4288185580 0 0) 0x7f7ebc19a040 con 0x7f7ebc107410 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.801+0000 7f7eb37fe700 1 -- 172.21.15.94:0/896413276 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc19a210 con 0x7f7ebc107410 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.801+0000 7f7ec4626700 1 -- 172.21.15.94:0/896413276 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ebc19a500 con 0x7f7ebc107410 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.801+0000 7f7ec4626700 1 -- 172.21.15.94:0/896413276 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ebc19e050 con 0x7f7ebc107410 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.801+0000 7f7eb37fe700 1 -- 172.21.15.94:0/896413276 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f7eb8001190 con 0x7f7ebc107410 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.801+0000 7f7eb37fe700 1 -- 172.21.15.94:0/896413276 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f7eb8005ee0 con 0x7f7ebc107410 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.801+0000 7f7eb37fe700 1 -- 172.21.15.94:0/896413276 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7eb8006510 con 0x7f7ebc107410 2024-03-12T03:08:19.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.801+0000 7f7eb37fe700 1 -- 172.21.15.94:0/896413276 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2600795447 0 0) 0x7f7eb8007c20 con 0x7f7ebc107410 2024-03-12T03:08:19.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.802+0000 7f7ec4626700 1 -- 172.21.15.94:0/896413276 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea0005150 con 0x7f7ebc107410 2024-03-12T03:08:19.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.804+0000 7f7eb37fe700 1 -- 172.21.15.94:0/896413276 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7eb8001c40 con 0x7f7ebc107410 2024-03-12T03:08:19.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.877+0000 7f7ec4626700 1 -- 172.21.15.94:0/896413276 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7ea0005440 con 0x7f7ebc107410 2024-03-12T03:08:19.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.877+0000 7f7eb37fe700 1 -- 172.21.15.94:0/896413276 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (unknown 4201413639 0 2115024113) 0x7f7eb8004ab0 con 0x7f7ebc107410 2024-03-12T03:08:19.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.878+0000 7f7ec4626700 1 -- 172.21.15.94:0/896413276 >> v1:172.21.15.94:6789/0 conn(0x7f7ebc107410 legacy=0x7f7ebc199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:19.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.878+0000 7f7ec4626700 1 -- 172.21.15.94:0/896413276 shutdown_connections 2024-03-12T03:08:19.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.878+0000 7f7ec4626700 1 -- 172.21.15.94:0/896413276 >> 172.21.15.94:0/896413276 conn(0x7f7ebc101290 msgr2=0x7f7ebc103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:19.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.878+0000 7f7ec4626700 1 -- 172.21.15.94:0/896413276 shutdown_connections 2024-03-12T03:08:19.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:18.878+0000 7f7ec4626700 1 -- 172.21.15.94:0/896413276 wait complete. 2024-03-12T03:08:19.622 INFO:teuthology.orchestra.run.smithi094.stdout:mgr not available, waiting (2/15)... 2024-03-12T03:08:19.777 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:19 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:19.495+0000 7f9edc763200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:08:20.035 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:19 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:19.774+0000 7f9edc763200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:08:20.035 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:20.032+0000 7f9edc763200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:08:20.459 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:20.261+0000 7f9edc763200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:08:21.459 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:21.049+0000 7f9edc763200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:08:21.937 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:21.610+0000 7f9edc763200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:08:21.937 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:21.710+0000 7f9edc763200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:08:21.937 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:21.934+0000 7f9edc763200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:08:22.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: Activating manager daemon y 2024-03-12T03:08:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: mgrmap e2: y(active, starting, since 0.00313922s) 2024-03-12T03:08:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.94:0/4193724842' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:08:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.94:0/4193724842' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:08:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.94:0/4193724842' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:08:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.94:0/4193724842' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:08:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.94:0/4193724842' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:08:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.94:0/4193724842' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:08:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.94:0/4193724842' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:08:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.94:0/4193724842' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:08:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: Manager daemon y is now available 2024-03-12T03:08:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.94:0/4193724842' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:08:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.94:0/4193724842' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:08:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.94:0/4193724842' entity='mgr.y' 2024-03-12T03:08:22.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.94:0/4193724842' entity='mgr.y' 2024-03-12T03:08:22.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:21 smithi094 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.94:0/4193724842' entity='mgr.y' 2024-03-12T03:08:22.746 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-03-12T03:08:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:08:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsid": "964142e8-e01d-11ee-95c8-87774f69a715", 2024-03-12T03:08:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:08:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:08:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:08:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:08:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:08:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:08:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:08:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:08:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:08:22.748 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T03:08:22.748 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:08:22.748 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-12T03:08:22.748 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:08:22.748 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:22.748 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T03:08:22.748 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:08:22.748 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:22.748 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:08:22.748 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:22.748 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:08:22.748 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:08:22.748 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:08:22.756 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:08:22.756 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:08:22.756 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:08:22.756 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:22.756 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:08:22.756 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:08:22.757 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:08:22.757 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:08:22.757 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:08:22.757 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:08:22.757 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:08:22.757 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:08:22.757 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:08:22.757 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:22.757 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:08:22.757 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:22.757 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:08:22.757 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:08:22.757 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:08:07.892677+0000", 2024-03-12T03:08:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.062+0000 7fcf63480700 1 Processor -- start 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.062+0000 7fcf63480700 1 -- start start 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.062+0000 7fcf63480700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcf5c10a160 con 0x7fcf5c105d30 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.062+0000 7fcf6121c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcf5c105d30 0x7fcf5c106150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36952/0 (socket says 172.21.15.94:36952) 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.062+0000 7fcf6121c700 1 -- 172.21.15.94:0/1642687780 learned_addr learned my addr 172.21.15.94:0/1642687780 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.062+0000 7fcf53fff700 1 -- 172.21.15.94:0/1642687780 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111640580 0 0) 0x7fcf5c10a160 con 0x7fcf5c105d30 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.062+0000 7fcf53fff700 1 -- 172.21.15.94:0/1642687780 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf480035b0 con 0x7fcf5c105d30 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.063+0000 7fcf53fff700 1 -- 172.21.15.94:0/1642687780 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2750893429 0 0) 0x7fcf480035b0 con 0x7fcf5c105d30 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.063+0000 7fcf53fff700 1 -- 172.21.15.94:0/1642687780 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c10b340 con 0x7fcf5c105d30 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.063+0000 7fcf53fff700 1 -- 172.21.15.94:0/1642687780 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fcf58001580 con 0x7fcf5c105d30 2024-03-12T03:08:22.759 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.063+0000 7fcf53fff700 1 -- 172.21.15.94:0/1642687780 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7fcf58001d20 con 0x7fcf5c105d30 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.063+0000 7fcf53fff700 1 -- 172.21.15.94:0/1642687780 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fcf58005960 con 0x7fcf5c105d30 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.063+0000 7fcf63480700 1 -- 172.21.15.94:0/1642687780 >> v1:172.21.15.94:6789/0 conn(0x7fcf5c105d30 legacy=0x7fcf5c106150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.063+0000 7fcf63480700 1 -- 172.21.15.94:0/1642687780 shutdown_connections 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.063+0000 7fcf63480700 1 -- 172.21.15.94:0/1642687780 >> 172.21.15.94:0/1642687780 conn(0x7fcf5c101290 msgr2=0x7fcf5c103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.063+0000 7fcf63480700 1 -- 172.21.15.94:0/1642687780 shutdown_connections 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.063+0000 7fcf63480700 1 -- 172.21.15.94:0/1642687780 wait complete. 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.064+0000 7fcf63480700 1 Processor -- start 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.064+0000 7fcf63480700 1 -- start start 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.064+0000 7fcf63480700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcf5c199e70 con 0x7fcf5c105d30 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.064+0000 7fcf6121c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcf5c105d30 0x7fcf5c199760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36968/0 (socket says 172.21.15.94:36968) 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.064+0000 7fcf6121c700 1 -- 172.21.15.94:0/2063915008 learned_addr learned my addr 172.21.15.94:0/2063915008 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.064+0000 7fcf527fc700 1 -- 172.21.15.94:0/2063915008 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171034162 0 0) 0x7fcf5c199e70 con 0x7fcf5c105d30 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.064+0000 7fcf527fc700 1 -- 172.21.15.94:0/2063915008 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf48005530 con 0x7fcf5c105d30 2024-03-12T03:08:22.760 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.064+0000 7fcf527fc700 1 -- 172.21.15.94:0/2063915008 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1720833711 0 0) 0x7fcf48005530 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.064+0000 7fcf527fc700 1 -- 172.21.15.94:0/2063915008 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf5c199e70 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.064+0000 7fcf527fc700 1 -- 172.21.15.94:0/2063915008 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fcf58001080 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.065+0000 7fcf527fc700 1 -- 172.21.15.94:0/2063915008 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2427679950 0 0) 0x7fcf5c199e70 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.065+0000 7fcf527fc700 1 -- 172.21.15.94:0/2063915008 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c19a040 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.065+0000 7fcf63480700 1 -- 172.21.15.94:0/2063915008 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf5c19a330 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.065+0000 7fcf63480700 1 -- 172.21.15.94:0/2063915008 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5c19de80 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.065+0000 7fcf527fc700 1 -- 172.21.15.94:0/2063915008 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7fcf58001cc0 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.065+0000 7fcf527fc700 1 -- 172.21.15.94:0/2063915008 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fcf58005f90 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.065+0000 7fcf527fc700 1 -- 172.21.15.94:0/2063915008 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 2) v1 ==== 45862+0+0 (unknown 240649462 0 0) 0x7fcf58011460 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.066+0000 7fcf527fc700 1 -- 172.21.15.94:0/2063915008 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2600795447 0 0) 0x7fcf58047fd0 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.066+0000 7fcf63480700 1 -- 172.21.15.94:0/2063915008 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf40005150 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.068+0000 7fcf527fc700 1 -- 172.21.15.94:0/2063915008 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcf580172d0 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.137+0000 7fcf63480700 1 -- 172.21.15.94:0/2063915008 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcf40005440 con 0x7fcf5c105d30 2024-03-12T03:08:22.761 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.138+0000 7fcf527fc700 1 -- 172.21.15.94:0/2063915008 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (unknown 4201413639 0 1019279604) 0x7fcf58016bd0 con 0x7fcf5c105d30 2024-03-12T03:08:22.762 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.138+0000 7fcf63480700 1 -- 172.21.15.94:0/2063915008 >> v1:172.21.15.94:6789/0 conn(0x7fcf5c105d30 legacy=0x7fcf5c199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:22.762 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.138+0000 7fcf63480700 1 -- 172.21.15.94:0/2063915008 shutdown_connections 2024-03-12T03:08:22.762 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.138+0000 7fcf63480700 1 -- 172.21.15.94:0/2063915008 >> 172.21.15.94:0/2063915008 conn(0x7fcf5c101290 msgr2=0x7fcf5c103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:22.762 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.138+0000 7fcf63480700 1 -- 172.21.15.94:0/2063915008 shutdown_connections 2024-03-12T03:08:22.762 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:22.138+0000 7fcf63480700 1 -- 172.21.15.94:0/2063915008 wait complete. 2024-03-12T03:08:22.762 INFO:teuthology.orchestra.run.smithi094.stdout:mgr not available, waiting (3/15)... 2024-03-12T03:08:23.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2063915008' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:08:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:22 smithi094 ceph-mon[29291]: mgrmap e3: y(active, since 1.00691s) 2024-03-12T03:08:25.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:25 smithi094 ceph-mon[29291]: mgrmap e4: y(active, since 2s) 2024-03-12T03:08:26.156 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1017645751' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:08:26.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-03-12T03:08:26.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:08:26.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsid": "964142e8-e01d-11ee-95c8-87774f69a715", 2024-03-12T03:08:26.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:08:26.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:08:26.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:08:26.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:08:26.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:26.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:08:26.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:08:26.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:08:26.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:08:26.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:08:26.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T03:08:26.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:08:26.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-12T03:08:26.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:08:26.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:26.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T03:08:26.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:08:26.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:26.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:08:26.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:26.163 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:08:26.163 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:08:26.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:08:26.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:08:26.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:08:26.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:08:26.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:26.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:08:26.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:08:26.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:08:26.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:08:26.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:08:07.892677+0000", 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:08:26.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.156+0000 7ff825772700 1 Processor -- start 2024-03-12T03:08:26.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.156+0000 7ff825772700 1 -- start start 2024-03-12T03:08:26.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.156+0000 7ff825772700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff820079670 con 0x7ff8201067e0 2024-03-12T03:08:26.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.157+0000 7ff81effd700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff8201067e0 0x7ff820108bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49108/0 (socket says 172.21.15.94:49108) 2024-03-12T03:08:26.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.157+0000 7ff81effd700 1 -- 172.21.15.94:0/1594906140 learned_addr learned my addr 172.21.15.94:0/1594906140 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:26.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.157+0000 7ff81e7fc700 1 -- 172.21.15.94:0/1594906140 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841667455 0 0) 0x7ff820079670 con 0x7ff8201067e0 2024-03-12T03:08:26.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.157+0000 7ff81e7fc700 1 -- 172.21.15.94:0/1594906140 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff80c0035b0 con 0x7ff8201067e0 2024-03-12T03:08:26.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.157+0000 7ff81e7fc700 1 -- 172.21.15.94:0/1594906140 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 371366992 0 0) 0x7ff80c0035b0 con 0x7ff8201067e0 2024-03-12T03:08:26.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.157+0000 7ff81e7fc700 1 -- 172.21.15.94:0/1594906140 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82010b300 con 0x7ff8201067e0 2024-03-12T03:08:26.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.157+0000 7ff81e7fc700 1 -- 172.21.15.94:0/1594906140 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7ff810001580 con 0x7ff8201067e0 2024-03-12T03:08:26.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.157+0000 7ff81e7fc700 1 -- 172.21.15.94:0/1594906140 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7ff810001d20 con 0x7ff8201067e0 2024-03-12T03:08:26.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.157+0000 7ff81e7fc700 1 -- 172.21.15.94:0/1594906140 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7ff810005960 con 0x7ff8201067e0 2024-03-12T03:08:26.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.158+0000 7ff825772700 1 -- 172.21.15.94:0/1594906140 >> v1:172.21.15.94:6789/0 conn(0x7ff8201067e0 legacy=0x7ff820108bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.158+0000 7ff825772700 1 -- 172.21.15.94:0/1594906140 shutdown_connections 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.158+0000 7ff825772700 1 -- 172.21.15.94:0/1594906140 >> 172.21.15.94:0/1594906140 conn(0x7ff8201001b0 msgr2=0x7ff8201025f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.158+0000 7ff825772700 1 -- 172.21.15.94:0/1594906140 shutdown_connections 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.158+0000 7ff825772700 1 -- 172.21.15.94:0/1594906140 wait complete. 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.158+0000 7ff825772700 1 Processor -- start 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.159+0000 7ff825772700 1 -- start start 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.159+0000 7ff825772700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff820199f10 con 0x7ff8201067e0 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.159+0000 7ff81effd700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff8201067e0 0x7ff820199800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49124/0 (socket says 172.21.15.94:49124) 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.159+0000 7ff81effd700 1 -- 172.21.15.94:0/1017645751 learned_addr learned my addr 172.21.15.94:0/1017645751 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.159+0000 7ff81cff9700 1 -- 172.21.15.94:0/1017645751 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765091449 0 0) 0x7ff820199f10 con 0x7ff8201067e0 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.159+0000 7ff81cff9700 1 -- 172.21.15.94:0/1017645751 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff80c005530 con 0x7ff8201067e0 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.159+0000 7ff81cff9700 1 -- 172.21.15.94:0/1017645751 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 228190268 0 0) 0x7ff80c005530 con 0x7ff8201067e0 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.159+0000 7ff81cff9700 1 -- 172.21.15.94:0/1017645751 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff820199f10 con 0x7ff8201067e0 2024-03-12T03:08:26.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.159+0000 7ff81cff9700 1 -- 172.21.15.94:0/1017645751 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7ff810004870 con 0x7ff8201067e0 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.160+0000 7ff81cff9700 1 -- 172.21.15.94:0/1017645751 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2868957722 0 0) 0x7ff820199f10 con 0x7ff8201067e0 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.160+0000 7ff81cff9700 1 -- 172.21.15.94:0/1017645751 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82019a0e0 con 0x7ff8201067e0 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.160+0000 7ff825772700 1 -- 172.21.15.94:0/1017645751 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff82019a3d0 con 0x7ff8201067e0 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.160+0000 7ff825772700 1 -- 172.21.15.94:0/1017645751 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82019df20 con 0x7ff8201067e0 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.160+0000 7ff81cff9700 1 -- 172.21.15.94:0/1017645751 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7ff810001190 con 0x7ff8201067e0 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.160+0000 7ff81cff9700 1 -- 172.21.15.94:0/1017645751 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7ff810005f70 con 0x7ff8201067e0 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.160+0000 7ff81cff9700 1 -- 172.21.15.94:0/1017645751 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 46060+0+0 (unknown 409760742 0 0) 0x7ff810011510 con 0x7ff8201067e0 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.161+0000 7ff825772700 1 -- 172.21.15.94:0/1017645751 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff800005120 con 0x7ff8201067e0 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.161+0000 7ff81cff9700 1 -- 172.21.15.94:0/1017645751 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2600795447 0 0) 0x7ff810048030 con 0x7ff8201067e0 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.166+0000 7ff81cff9700 1 -- 172.21.15.94:0/1017645751 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7ff8100173d0 con 0x7ff8201067e0 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.436+0000 7ff825772700 1 -- 172.21.15.94:0/1017645751 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff800005410 con 0x7ff8201067e0 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.436+0000 7ff81cff9700 1 -- 172.21.15.94:0/1017645751 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (unknown 4201413639 0 1112538742) 0x7ff810016cd0 con 0x7ff8201067e0 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.438+0000 7ff825772700 1 -- 172.21.15.94:0/1017645751 >> v1:172.21.15.94:6800/2955156665 conn(0x7ff80c03a1f0 legacy=0x7ff80c03c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:26.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.438+0000 7ff825772700 1 -- 172.21.15.94:0/1017645751 >> v1:172.21.15.94:6789/0 conn(0x7ff8201067e0 legacy=0x7ff820199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:26.177 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.438+0000 7ff825772700 1 -- 172.21.15.94:0/1017645751 shutdown_connections 2024-03-12T03:08:26.177 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.438+0000 7ff825772700 1 -- 172.21.15.94:0/1017645751 >> 172.21.15.94:0/1017645751 conn(0x7ff8201001b0 msgr2=0x7ff820190470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:26.177 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.438+0000 7ff825772700 1 -- 172.21.15.94:0/1017645751 shutdown_connections 2024-03-12T03:08:26.177 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:25.438+0000 7ff825772700 1 -- 172.21.15.94:0/1017645751 wait complete. 2024-03-12T03:08:26.177 INFO:teuthology.orchestra.run.smithi094.stdout:mgr is available 2024-03-12T03:08:27.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1750594740' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-12T03:08:27.328 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-03-12T03:08:27.328 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout fsid = 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.578+0000 7f7623a75700 1 Processor -- start 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.578+0000 7f7623a75700 1 -- start start 2024-03-12T03:08:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.578+0000 7f7623a75700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f761c10a190 con 0x7f761c105d60 2024-03-12T03:08:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.578+0000 7f7621811700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f761c105d60 0x7f761c106180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49126/0 (socket says 172.21.15.94:49126) 2024-03-12T03:08:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.578+0000 7f7621811700 1 -- 172.21.15.94:0/2474977783 learned_addr learned my addr 172.21.15.94:0/2474977783 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.579+0000 7f762080f700 1 -- 172.21.15.94:0/2474977783 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670077688 0 0) 0x7f761c10a190 con 0x7f761c105d60 2024-03-12T03:08:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.579+0000 7f762080f700 1 -- 172.21.15.94:0/2474977783 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76080035b0 con 0x7f761c105d60 2024-03-12T03:08:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.579+0000 7f762080f700 1 -- 172.21.15.94:0/2474977783 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2351649459 0 0) 0x7f76080035b0 con 0x7f761c105d60 2024-03-12T03:08:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.579+0000 7f762080f700 1 -- 172.21.15.94:0/2474977783 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c10b370 con 0x7f761c105d60 2024-03-12T03:08:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.579+0000 7f762080f700 1 -- 172.21.15.94:0/2474977783 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f7618001580 con 0x7f761c105d60 2024-03-12T03:08:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.579+0000 7f762080f700 1 -- 172.21.15.94:0/2474977783 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f7618001d20 con 0x7f761c105d60 2024-03-12T03:08:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.579+0000 7f762080f700 1 -- 172.21.15.94:0/2474977783 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f7618005960 con 0x7f761c105d60 2024-03-12T03:08:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.579+0000 7f7623a75700 1 -- 172.21.15.94:0/2474977783 >> v1:172.21.15.94:6789/0 conn(0x7f761c105d60 legacy=0x7f761c106180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.580+0000 7f7623a75700 1 -- 172.21.15.94:0/2474977783 shutdown_connections 2024-03-12T03:08:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.580+0000 7f7623a75700 1 -- 172.21.15.94:0/2474977783 >> 172.21.15.94:0/2474977783 conn(0x7f761c1012e0 msgr2=0x7f761c103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.580+0000 7f7623a75700 1 -- 172.21.15.94:0/2474977783 shutdown_connections 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.580+0000 7f7623a75700 1 -- 172.21.15.94:0/2474977783 wait complete. 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.580+0000 7f7623a75700 1 Processor -- start 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.580+0000 7f7623a75700 1 -- start start 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.580+0000 7f7623a75700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f761c199f20 con 0x7f761c105d60 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.580+0000 7f7621811700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f761c105d60 0x7f761c199810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49128/0 (socket says 172.21.15.94:49128) 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.580+0000 7f7621811700 1 -- 172.21.15.94:0/1750594740 learned_addr learned my addr 172.21.15.94:0/1750594740 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.581+0000 7f7612ffd700 1 -- 172.21.15.94:0/1750594740 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601004535 0 0) 0x7f761c199f20 con 0x7f761c105d60 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.581+0000 7f7612ffd700 1 -- 172.21.15.94:0/1750594740 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7608005530 con 0x7f761c105d60 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.581+0000 7f7612ffd700 1 -- 172.21.15.94:0/1750594740 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2616937845 0 0) 0x7f7608005530 con 0x7f761c105d60 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.581+0000 7f7612ffd700 1 -- 172.21.15.94:0/1750594740 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f761c199f20 con 0x7f761c105d60 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.581+0000 7f7612ffd700 1 -- 172.21.15.94:0/1750594740 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f7618004870 con 0x7f761c105d60 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.581+0000 7f7612ffd700 1 -- 172.21.15.94:0/1750594740 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 401444617 0 0) 0x7f761c199f20 con 0x7f761c105d60 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.581+0000 7f7612ffd700 1 -- 172.21.15.94:0/1750594740 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c19a0f0 con 0x7f761c105d60 2024-03-12T03:08:27.331 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.581+0000 7f7623a75700 1 -- 172.21.15.94:0/1750594740 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c19a3e0 con 0x7f761c105d60 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.581+0000 7f7623a75700 1 -- 172.21.15.94:0/1750594740 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c19df30 con 0x7f761c105d60 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.581+0000 7f7612ffd700 1 -- 172.21.15.94:0/1750594740 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f7618004a40 con 0x7f761c105d60 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.581+0000 7f7612ffd700 1 -- 172.21.15.94:0/1750594740 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f7618005ea0 con 0x7f761c105d60 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.582+0000 7f7612ffd700 1 -- 172.21.15.94:0/1750594740 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 46060+0+0 (unknown 409760742 0 0) 0x7f7618007170 con 0x7f761c105d60 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.582+0000 7f7623a75700 1 -- 172.21.15.94:0/1750594740 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7600005150 con 0x7f761c105d60 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.582+0000 7f7612ffd700 1 -- 172.21.15.94:0/1750594740 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2600795447 0 0) 0x7f76180474f0 con 0x7f761c105d60 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.587+0000 7f7612ffd700 1 -- 172.21.15.94:0/1750594740 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f7618017340 con 0x7f761c105d60 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.777+0000 7f7623a75700 1 -- 172.21.15.94:0/1750594740 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f7600002430 con 0x7f761c105d60 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.779+0000 7f7612ffd700 1 -- 172.21.15.94:0/1750594740 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+282 (unknown 1187553405 0 2939638839) 0x7f7618016c40 con 0x7f761c105d60 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.780+0000 7f7623a75700 1 -- 172.21.15.94:0/1750594740 >> v1:172.21.15.94:6800/2955156665 conn(0x7f760803a1a0 legacy=0x7f760803c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.780+0000 7f7623a75700 1 -- 172.21.15.94:0/1750594740 >> v1:172.21.15.94:6789/0 conn(0x7f761c105d60 legacy=0x7f761c199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.781+0000 7f7623a75700 1 -- 172.21.15.94:0/1750594740 shutdown_connections 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.781+0000 7f7623a75700 1 -- 172.21.15.94:0/1750594740 >> 172.21.15.94:0/1750594740 conn(0x7f761c1012e0 msgr2=0x7f761c103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:27.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.781+0000 7f7623a75700 1 -- 172.21.15.94:0/1750594740 shutdown_connections 2024-03-12T03:08:27.333 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:26.781+0000 7f7623a75700 1 -- 172.21.15.94:0/1750594740 wait complete. 2024-03-12T03:08:27.333 INFO:teuthology.orchestra.run.smithi094.stdout:Enabling cephadm module... 2024-03-12T03:08:28.318 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1249490921' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-12T03:08:28.318 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1249490921' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-12T03:08:28.318 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:28 smithi094 ceph-mon[29291]: mgrmap e5: y(active, since 6s) 2024-03-12T03:08:28.318 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:28 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ignoring --setuser ceph since I am not root 2024-03-12T03:08:28.318 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:28 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ignoring --setgroup ceph since I am not root 2024-03-12T03:08:28.592 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.744+0000 7f9da75c1700 1 Processor -- start 2024-03-12T03:08:28.592 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.744+0000 7f9da75c1700 1 -- start start 2024-03-12T03:08:28.592 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.744+0000 7f9da75c1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9da0109040 con 0x7f9da0104c10 2024-03-12T03:08:28.592 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.745+0000 7f9da535d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9da0104c10 0x7f9da0105030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49132/0 (socket says 172.21.15.94:49132) 2024-03-12T03:08:28.592 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.745+0000 7f9da535d700 1 -- 172.21.15.94:0/4007354819 learned_addr learned my addr 172.21.15.94:0/4007354819 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:28.592 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.745+0000 7f9d97fff700 1 -- 172.21.15.94:0/4007354819 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047564020 0 0) 0x7f9da0109040 con 0x7f9da0104c10 2024-03-12T03:08:28.592 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.745+0000 7f9d97fff700 1 -- 172.21.15.94:0/4007354819 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c0035b0 con 0x7f9da0104c10 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.745+0000 7f9d97fff700 1 -- 172.21.15.94:0/4007354819 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 830309735 0 0) 0x7f9d8c0035b0 con 0x7f9da0104c10 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.745+0000 7f9d97fff700 1 -- 172.21.15.94:0/4007354819 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da010a220 con 0x7f9da0104c10 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.745+0000 7f9d97fff700 1 -- 172.21.15.94:0/4007354819 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f9d9c001580 con 0x7f9da0104c10 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.745+0000 7f9d97fff700 1 -- 172.21.15.94:0/4007354819 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f9d9c001d20 con 0x7f9da0104c10 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.745+0000 7f9d97fff700 1 -- 172.21.15.94:0/4007354819 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f9d9c005960 con 0x7f9da0104c10 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.746+0000 7f9da75c1700 1 -- 172.21.15.94:0/4007354819 >> v1:172.21.15.94:6789/0 conn(0x7f9da0104c10 legacy=0x7f9da0105030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.746+0000 7f9da75c1700 1 -- 172.21.15.94:0/4007354819 shutdown_connections 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.746+0000 7f9da75c1700 1 -- 172.21.15.94:0/4007354819 >> 172.21.15.94:0/4007354819 conn(0x7f9da01001b0 msgr2=0x7f9da01025f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.746+0000 7f9da75c1700 1 -- 172.21.15.94:0/4007354819 shutdown_connections 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.746+0000 7f9da75c1700 1 -- 172.21.15.94:0/4007354819 wait complete. 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.746+0000 7f9da75c1700 1 Processor -- start 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.746+0000 7f9da75c1700 1 -- start start 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.746+0000 7f9da75c1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9da0079270 con 0x7f9da0104c10 2024-03-12T03:08:28.593 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.747+0000 7f9da535d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9da0104c10 0x7f9da019c530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49142/0 (socket says 172.21.15.94:49142) 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.747+0000 7f9da535d700 1 -- 172.21.15.94:0/1249490921 learned_addr learned my addr 172.21.15.94:0/1249490921 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.747+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843368064 0 0) 0x7f9da0079270 con 0x7f9da0104c10 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.747+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c005530 con 0x7f9da0104c10 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.747+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1054522185 0 0) 0x7f9d8c005530 con 0x7f9da0104c10 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.747+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9da0079270 con 0x7f9da0104c10 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.747+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f9d9c001080 con 0x7f9da0104c10 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.747+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2165472149 0 0) 0x7f9da0079270 con 0x7f9da0104c10 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.747+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da0079440 con 0x7f9da0104c10 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.747+0000 7f9da75c1700 1 -- 172.21.15.94:0/1249490921 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da0079730 con 0x7f9da0104c10 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.747+0000 7f9da75c1700 1 -- 172.21.15.94:0/1249490921 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da019df00 con 0x7f9da0104c10 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.747+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f9d9c001cc0 con 0x7f9da0104c10 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.747+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f9d9c005f90 con 0x7f9da0104c10 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.748+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 46060+0+0 (unknown 409760742 0 0) 0x7f9d9c011530 con 0x7f9da0104c10 2024-03-12T03:08:28.594 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.748+0000 7f9da75c1700 1 -- 172.21.15.94:0/1249490921 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d84005150 con 0x7f9da0104c10 2024-03-12T03:08:28.595 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.748+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2600795447 0 0) 0x7f9d9c0475b0 con 0x7f9da0104c10 2024-03-12T03:08:28.595 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:27.753+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f9d9c0173c0 con 0x7f9da0104c10 2024-03-12T03:08:28.595 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:28.001+0000 7f9da75c1700 1 -- 172.21.15.94:0/1249490921 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f9d84005440 con 0x7f9da0104c10 2024-03-12T03:08:28.595 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:28.002+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (unknown 2263024820 0 0) 0x7f9d9c016cc0 con 0x7f9da0104c10 2024-03-12T03:08:28.595 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:28.004+0000 7f9d967fc700 1 -- 172.21.15.94:0/1249490921 <== mon.0 v1:172.21.15.94:6789/0 11 ==== mgrmap(e 5) v1 ==== 46071+0+0 (unknown 4051996124 0 0) 0x7f9d9c045fa0 con 0x7f9da0104c10 2024-03-12T03:08:28.595 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:28.005+0000 7f9da75c1700 1 -- 172.21.15.94:0/1249490921 >> v1:172.21.15.94:6800/2955156665 conn(0x7f9d8c03a340 legacy=0x7f9d8c03c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:28.595 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:28.005+0000 7f9da75c1700 1 -- 172.21.15.94:0/1249490921 >> v1:172.21.15.94:6789/0 conn(0x7f9da0104c10 legacy=0x7f9da019c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:28.595 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:28.005+0000 7f9da75c1700 1 -- 172.21.15.94:0/1249490921 shutdown_connections 2024-03-12T03:08:28.595 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:28.005+0000 7f9da75c1700 1 -- 172.21.15.94:0/1249490921 >> 172.21.15.94:0/1249490921 conn(0x7f9da01001b0 msgr2=0x7f9da01025f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:28.595 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:28.005+0000 7f9da75c1700 1 -- 172.21.15.94:0/1249490921 shutdown_connections 2024-03-12T03:08:28.595 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:28.005+0000 7f9da75c1700 1 -- 172.21.15.94:0/1249490921 wait complete. 2024-03-12T03:08:28.834 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:28 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:28.569+0000 7ff01c444200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:08:28.835 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:28 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:28.670+0000 7ff01c444200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:08:29.565 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1268971612' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T03:08:29.565 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:29 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:29.326+0000 7ff01c444200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:08:29.565 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:29 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:29.422+0000 7ff01c444200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:08:29.565 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:29 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:29.562+0000 7ff01c444200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:08:29.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:29 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:29.657+0000 7ff01c444200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:08:29.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:29 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:29.742+0000 7ff01c444200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:08:30.057 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:08:30.057 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-12T03:08:30.057 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T03:08:30.057 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-12T03:08:30.057 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T03:08:30.057 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:08:30.057 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.042+0000 7fa87f7e1700 1 Processor -- start 2024-03-12T03:08:30.058 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.042+0000 7fa87f7e1700 1 -- start start 2024-03-12T03:08:30.058 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.042+0000 7fa87f7e1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa878078e00 con 0x7fa8781067b0 2024-03-12T03:08:30.058 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.042+0000 7fa87d57d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa8781067b0 0x7fa878108ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49164/0 (socket says 172.21.15.94:49164) 2024-03-12T03:08:30.058 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.042+0000 7fa87d57d700 1 -- 172.21.15.94:0/1997341509 learned_addr learned my addr 172.21.15.94:0/1997341509 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:30.058 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.043+0000 7fa86ffff700 1 -- 172.21.15.94:0/1997341509 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935870718 0 0) 0x7fa878078e00 con 0x7fa8781067b0 2024-03-12T03:08:30.058 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.043+0000 7fa86ffff700 1 -- 172.21.15.94:0/1997341509 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8640035b0 con 0x7fa8781067b0 2024-03-12T03:08:30.058 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.043+0000 7fa86ffff700 1 -- 172.21.15.94:0/1997341509 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 337030291 0 0) 0x7fa8640035b0 con 0x7fa8781067b0 2024-03-12T03:08:30.058 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.043+0000 7fa86ffff700 1 -- 172.21.15.94:0/1997341509 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87810b2d0 con 0x7fa8781067b0 2024-03-12T03:08:30.058 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.043+0000 7fa86ffff700 1 -- 172.21.15.94:0/1997341509 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fa874001580 con 0x7fa8781067b0 2024-03-12T03:08:30.058 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.043+0000 7fa86ffff700 1 -- 172.21.15.94:0/1997341509 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7fa874001d20 con 0x7fa8781067b0 2024-03-12T03:08:30.058 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.043+0000 7fa86ffff700 1 -- 172.21.15.94:0/1997341509 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fa874005960 con 0x7fa8781067b0 2024-03-12T03:08:30.058 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.044+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1997341509 >> v1:172.21.15.94:6789/0 conn(0x7fa8781067b0 legacy=0x7fa878108ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.044+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1997341509 shutdown_connections 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.044+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1997341509 >> 172.21.15.94:0/1997341509 conn(0x7fa878100160 msgr2=0x7fa8781025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.044+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1997341509 shutdown_connections 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.044+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1997341509 wait complete. 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.045+0000 7fa87f7e1700 1 Processor -- start 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.045+0000 7fa87f7e1700 1 -- start start 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.045+0000 7fa87f7e1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa878199ee0 con 0x7fa8781067b0 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.045+0000 7fa87d57d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa8781067b0 0x7fa8781997d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49168/0 (socket says 172.21.15.94:49168) 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.045+0000 7fa87d57d700 1 -- 172.21.15.94:0/1268971612 learned_addr learned my addr 172.21.15.94:0/1268971612 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.045+0000 7fa86e7fc700 1 -- 172.21.15.94:0/1268971612 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124973676 0 0) 0x7fa878199ee0 con 0x7fa8781067b0 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.045+0000 7fa86e7fc700 1 -- 172.21.15.94:0/1268971612 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa864005530 con 0x7fa8781067b0 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.045+0000 7fa86e7fc700 1 -- 172.21.15.94:0/1268971612 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 72240946 0 0) 0x7fa864005530 con 0x7fa8781067b0 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.045+0000 7fa86e7fc700 1 -- 172.21.15.94:0/1268971612 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa878199ee0 con 0x7fa8781067b0 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.045+0000 7fa86e7fc700 1 -- 172.21.15.94:0/1268971612 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fa874004870 con 0x7fa8781067b0 2024-03-12T03:08:30.059 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.046+0000 7fa86e7fc700 1 -- 172.21.15.94:0/1268971612 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1094402711 0 0) 0x7fa878199ee0 con 0x7fa8781067b0 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.046+0000 7fa86e7fc700 1 -- 172.21.15.94:0/1268971612 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87819a0b0 con 0x7fa8781067b0 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.046+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1268971612 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87819a3a0 con 0x7fa8781067b0 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.046+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1268971612 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87819def0 con 0x7fa8781067b0 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.046+0000 7fa86e7fc700 1 -- 172.21.15.94:0/1268971612 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7fa874001ce0 con 0x7fa8781067b0 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.046+0000 7fa86e7fc700 1 -- 172.21.15.94:0/1268971612 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fa874006070 con 0x7fa8781067b0 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.047+0000 7fa86e7fc700 1 -- 172.21.15.94:0/1268971612 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 5) v1 ==== 46071+0+0 (unknown 4051996124 0 0) 0x7fa8740116d0 con 0x7fa8781067b0 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.047+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1268971612 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85c005120 con 0x7fa8781067b0 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.047+0000 7fa87cd7c700 1 -- 172.21.15.94:0/1268971612 >> v1:172.21.15.94:6800/2955156665 conn(0x7fa86403a1a0 legacy=0x7fa86403c660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/2955156665 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.047+0000 7fa86e7fc700 1 -- 172.21.15.94:0/1268971612 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2600795447 0 0) 0x7fa8740477c0 con 0x7fa8781067b0 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.051+0000 7fa86e7fc700 1 -- 172.21.15.94:0/1268971612 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fa8740175d0 con 0x7fa8781067b0 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.248+0000 7fa87cd7c700 1 -- 172.21.15.94:0/1268971612 >> v1:172.21.15.94:6800/2955156665 conn(0x7fa86403a1a0 legacy=0x7fa86403c660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/2955156665 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.312+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1268971612 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa85c005c20 con 0x7fa8781067b0 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.312+0000 7fa86e7fc700 1 -- 172.21.15.94:0/1268971612 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (unknown 3768197548 0 15966916) 0x7fa874016ed0 con 0x7fa8781067b0 2024-03-12T03:08:30.060 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.315+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1268971612 >> v1:172.21.15.94:6800/2955156665 conn(0x7fa86403a1a0 legacy=0x7fa86403c660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T03:08:30.061 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.315+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1268971612 >> v1:172.21.15.94:6789/0 conn(0x7fa8781067b0 legacy=0x7fa8781997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:30.061 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.315+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1268971612 shutdown_connections 2024-03-12T03:08:30.061 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.315+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1268971612 >> 172.21.15.94:0/1268971612 conn(0x7fa878100160 msgr2=0x7fa878190460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:30.061 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.315+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1268971612 shutdown_connections 2024-03-12T03:08:30.061 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:29.315+0000 7fa87f7e1700 1 -- 172.21.15.94:0/1268971612 wait complete. 2024-03-12T03:08:30.061 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for the mgr to restart... 2024-03-12T03:08:30.061 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mgr epoch 5... 2024-03-12T03:08:30.259 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:30.005+0000 7ff01c444200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:08:30.260 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:30.123+0000 7ff01c444200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:08:30.260 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:30.213+0000 7ff01c444200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:08:30.668 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:30.312+0000 7ff01c444200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:08:30.669 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:30.492+0000 7ff01c444200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:08:31.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:31 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:31.445+0000 7ff01c444200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:08:31.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:31 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:31.576+0000 7ff01c444200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:08:32.209 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:31 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:31.924+0000 7ff01c444200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:08:32.209 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:32 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:32.013+0000 7ff01c444200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:08:32.209 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:32 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:32.206+0000 7ff01c444200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:08:32.459 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:32 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:32.389+0000 7ff01c444200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:08:32.943 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:32 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:32.670+0000 7ff01c444200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:08:32.944 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:32 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:32.940+0000 7ff01c444200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:08:33.209 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:33 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:33.174+0000 7ff01c444200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:08:34.459 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:33 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:33.962+0000 7ff01c444200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:08:34.840 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:34.525+0000 7ff01c444200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:08:34.840 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:34.621+0000 7ff01c444200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:08:34.840 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:34.837+0000 7ff01c444200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:08:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: Active manager daemon y restarted 2024-03-12T03:08:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: Activating manager daemon y 2024-03-12T03:08:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: osdmap e2: 0 total, 0 up, 0 in 2024-03-12T03:08:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: mgrmap e6: y(active, starting, since 0.00313148s) 2024-03-12T03:08:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:08:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:08:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:08:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:08:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:08:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: Manager daemon y is now available 2024-03-12T03:08:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:08:35.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:08:35.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:34 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:08:36.181 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:35 smithi094 ceph-mon[29291]: Found migration_current of "None". Setting to last migration. 2024-03-12T03:08:36.181 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:35 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:08:36.181 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:35 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:08:36.181 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:35 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:36.181 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:35 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:36.181 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:35 smithi094 ceph-mon[29291]: mgrmap e7: y(active, since 1.00564s) 2024-03-12T03:08:36.464 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:08:36.464 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-12T03:08:36.464 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T03:08:36.464 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:08:36.464 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.460+0000 7f41b8c72700 1 Processor -- start 2024-03-12T03:08:36.464 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.460+0000 7f41b8c72700 1 -- start start 2024-03-12T03:08:36.464 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.460+0000 7f41b8c72700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f41b4078dc0 con 0x7f41b41045e0 2024-03-12T03:08:36.464 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.460+0000 7f41b259c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f41b41045e0 0x7f41b4104a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49178/0 (socket says 172.21.15.94:49178) 2024-03-12T03:08:36.464 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.460+0000 7f41b259c700 1 -- 172.21.15.94:0/2777136227 learned_addr learned my addr 172.21.15.94:0/2777136227 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:36.464 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.461+0000 7f41b159a700 1 -- 172.21.15.94:0/2777136227 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124303793 0 0) 0x7f41b4078dc0 con 0x7f41b41045e0 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.461+0000 7f41b159a700 1 -- 172.21.15.94:0/2777136227 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f419c0035b0 con 0x7f41b41045e0 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.461+0000 7f41b159a700 1 -- 172.21.15.94:0/2777136227 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3530446053 0 0) 0x7f419c0035b0 con 0x7f41b41045e0 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.461+0000 7f41b159a700 1 -- 172.21.15.94:0/2777136227 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b4106120 con 0x7f41b41045e0 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.461+0000 7f41b159a700 1 -- 172.21.15.94:0/2777136227 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f41a8001580 con 0x7f41b41045e0 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.461+0000 7f41b159a700 1 -- 172.21.15.94:0/2777136227 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f41a8001d20 con 0x7f41b41045e0 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.461+0000 7f41b159a700 1 -- 172.21.15.94:0/2777136227 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f41a8005960 con 0x7f41b41045e0 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.462+0000 7f41b8c72700 1 -- 172.21.15.94:0/2777136227 >> v1:172.21.15.94:6789/0 conn(0x7f41b41045e0 legacy=0x7f41b4104a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.462+0000 7f41b8c72700 1 -- 172.21.15.94:0/2777136227 shutdown_connections 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.462+0000 7f41b8c72700 1 -- 172.21.15.94:0/2777136227 >> 172.21.15.94:0/2777136227 conn(0x7f41b4100160 msgr2=0x7f41b41025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.462+0000 7f41b8c72700 1 -- 172.21.15.94:0/2777136227 shutdown_connections 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.462+0000 7f41b8c72700 1 -- 172.21.15.94:0/2777136227 wait complete. 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.463+0000 7f41b8c72700 1 Processor -- start 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.463+0000 7f41b8c72700 1 -- start start 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.463+0000 7f41b8c72700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f41b419c0f0 con 0x7f41b41045e0 2024-03-12T03:08:36.465 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.463+0000 7f41b259c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f41b41045e0 0x7f41b419b9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49186/0 (socket says 172.21.15.94:49186) 2024-03-12T03:08:36.466 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.463+0000 7f41b259c700 1 -- 172.21.15.94:0/151662371 learned_addr learned my addr 172.21.15.94:0/151662371 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:36.466 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.463+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610070115 0 0) 0x7f41b419c0f0 con 0x7f41b41045e0 2024-03-12T03:08:36.466 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.463+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f419c005530 con 0x7f41b41045e0 2024-03-12T03:08:36.466 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.463+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2824639640 0 0) 0x7f419c005530 con 0x7f41b41045e0 2024-03-12T03:08:36.466 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.463+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41b419c0f0 con 0x7f41b41045e0 2024-03-12T03:08:36.466 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.463+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f41a8004870 con 0x7f41b41045e0 2024-03-12T03:08:36.466 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.464+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3480203091 0 0) 0x7f41b419c0f0 con 0x7f41b41045e0 2024-03-12T03:08:36.466 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.464+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b419c2c0 con 0x7f41b41045e0 2024-03-12T03:08:36.466 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.464+0000 7f41b8c72700 1 -- 172.21.15.94:0/151662371 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b419c5b0 con 0x7f41b41045e0 2024-03-12T03:08:36.466 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.464+0000 7f41b8c72700 1 -- 172.21.15.94:0/151662371 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b41a0100 con 0x7f41b41045e0 2024-03-12T03:08:36.466 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.464+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f41a8001ce0 con 0x7f41b41045e0 2024-03-12T03:08:36.466 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.464+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f41a80060c0 con 0x7f41b41045e0 2024-03-12T03:08:36.466 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.465+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 5) v1 ==== 46071+0+0 (unknown 4051996124 0 0) 0x7f41a8011660 con 0x7f41b41045e0 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.465+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 --> v1:172.21.15.94:6800/2955156665 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f419c03dec0 con 0x7f419c03a1f0 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.465+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2600795447 0 0) 0x7f41a80484a0 con 0x7f41b41045e0 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.465+0000 7f41b1d9b700 1 -- 172.21.15.94:0/151662371 >> v1:172.21.15.94:6800/2955156665 conn(0x7f419c03a1f0 legacy=0x7f419c03c6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/2955156665 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:30.666+0000 7f41b1d9b700 1 -- 172.21.15.94:0/151662371 >> v1:172.21.15.94:6800/2955156665 conn(0x7f419c03a1f0 legacy=0x7f419c03c6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/2955156665 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:31.067+0000 7f41b1d9b700 1 -- 172.21.15.94:0/151662371 >> v1:172.21.15.94:6800/2955156665 conn(0x7f419c03a1f0 legacy=0x7f419c03c6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/2955156665 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:31.868+0000 7f41b1d9b700 1 -- 172.21.15.94:0/151662371 >> v1:172.21.15.94:6800/2955156665 conn(0x7f419c03a1f0 legacy=0x7f419c03c6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/2955156665 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:33.469+0000 7f41b1d9b700 1 -- 172.21.15.94:0/151662371 >> v1:172.21.15.94:6800/2955156665 conn(0x7f419c03a1f0 legacy=0x7f419c03c6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/2955156665 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:34.851+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mgrmap(e 6) v1 ==== 45873+0+0 (unknown 1624182372 0 0) 0x7f41a8013460 con 0x7f41b41045e0 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:34.851+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 >> v1:172.21.15.94:6800/2955156665 conn(0x7f419c03a1f0 legacy=0x7f419c03c6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:35.853+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mgrmap(e 7) v1 ==== 45965+0+0 (unknown 4271366492 0 0) 0x7f41a8049140 con 0x7f41b41045e0 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:35.853+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 --> v1:172.21.15.94:6800/3548280253 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f419c03dec0 con 0x7f419c03ec10 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:35.854+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 <== mgr.14120 v1:172.21.15.94:6800/3548280253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (unknown 0 0 4109177037) 0x7f419c03dec0 con 0x7f419c03ec10 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:35.861+0000 7f41b8c72700 1 -- 172.21.15.94:0/151662371 --> v1:172.21.15.94:6800/3548280253 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4194002630 con 0x7f419c03ec10 2024-03-12T03:08:36.467 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:35.861+0000 7f41a37fe700 1 -- 172.21.15.94:0/151662371 <== mgr.14120 v1:172.21.15.94:6800/3548280253 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (unknown 0 0 96372106) 0x7f4194002630 con 0x7f419c03ec10 2024-03-12T03:08:36.468 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:35.861+0000 7f41b8c72700 1 -- 172.21.15.94:0/151662371 >> v1:172.21.15.94:6800/3548280253 conn(0x7f419c03ec10 legacy=0x7f419c041000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:36.468 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:35.861+0000 7f41b8c72700 1 -- 172.21.15.94:0/151662371 >> v1:172.21.15.94:6789/0 conn(0x7f41b41045e0 legacy=0x7f41b419b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:36.468 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:35.861+0000 7f41b8c72700 1 -- 172.21.15.94:0/151662371 shutdown_connections 2024-03-12T03:08:36.468 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:35.861+0000 7f41b8c72700 1 -- 172.21.15.94:0/151662371 >> 172.21.15.94:0/151662371 conn(0x7f41b4100160 msgr2=0x7f41b41025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:36.468 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:35.861+0000 7f41b8c72700 1 -- 172.21.15.94:0/151662371 shutdown_connections 2024-03-12T03:08:36.468 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:35.861+0000 7f41b8c72700 1 -- 172.21.15.94:0/151662371 wait complete. 2024-03-12T03:08:36.468 INFO:teuthology.orchestra.run.smithi094.stdout:mgr epoch 5 is available 2024-03-12T03:08:36.468 INFO:teuthology.orchestra.run.smithi094.stdout:Setting orchestrator backend to cephadm... 2024-03-12T03:08:37.113 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:36 smithi094 ceph-mon[29291]: from='client.14124 v1:172.21.15.94:0/151662371' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T03:08:37.113 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:36 smithi094 ceph-mon[29291]: from='client.14124 v1:172.21.15.94:0/151662371' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T03:08:37.113 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:36 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:08:37.695 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.889+0000 7f2912011700 1 Processor -- start 2024-03-12T03:08:37.695 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.889+0000 7f2912011700 1 -- start start 2024-03-12T03:08:37.695 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.889+0000 7f2912011700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f290c10a130 con 0x7f290c105d00 2024-03-12T03:08:37.695 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.890+0000 7f290b7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f290c105d00 0x7f290c106120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36298/0 (socket says 172.21.15.94:36298) 2024-03-12T03:08:37.695 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.890+0000 7f290b7fe700 1 -- 172.21.15.94:0/3741451212 learned_addr learned my addr 172.21.15.94:0/3741451212 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:37.695 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.890+0000 7f290affd700 1 -- 172.21.15.94:0/3741451212 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493098169 0 0) 0x7f290c10a130 con 0x7f290c105d00 2024-03-12T03:08:37.695 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.890+0000 7f290affd700 1 -- 172.21.15.94:0/3741451212 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f80035b0 con 0x7f290c105d00 2024-03-12T03:08:37.695 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.890+0000 7f290affd700 1 -- 172.21.15.94:0/3741451212 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2747843406 0 0) 0x7f28f80035b0 con 0x7f290c105d00 2024-03-12T03:08:37.695 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.890+0000 7f290affd700 1 -- 172.21.15.94:0/3741451212 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c10b310 con 0x7f290c105d00 2024-03-12T03:08:37.695 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.890+0000 7f290affd700 1 -- 172.21.15.94:0/3741451212 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f28fc001580 con 0x7f290c105d00 2024-03-12T03:08:37.695 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.890+0000 7f290affd700 1 -- 172.21.15.94:0/3741451212 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f28fc001d20 con 0x7f290c105d00 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.890+0000 7f290affd700 1 -- 172.21.15.94:0/3741451212 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f28fc005960 con 0x7f290c105d00 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.891+0000 7f2912011700 1 -- 172.21.15.94:0/3741451212 >> v1:172.21.15.94:6789/0 conn(0x7f290c105d00 legacy=0x7f290c106120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.891+0000 7f2912011700 1 -- 172.21.15.94:0/3741451212 shutdown_connections 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.891+0000 7f2912011700 1 -- 172.21.15.94:0/3741451212 >> 172.21.15.94:0/3741451212 conn(0x7f290c1012a0 msgr2=0x7f290c1036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.891+0000 7f2912011700 1 -- 172.21.15.94:0/3741451212 shutdown_connections 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.891+0000 7f2912011700 1 -- 172.21.15.94:0/3741451212 wait complete. 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.891+0000 7f2912011700 1 Processor -- start 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.891+0000 7f2912011700 1 -- start start 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.892+0000 7f2912011700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f290c199ee0 con 0x7f290c105d00 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.892+0000 7f290b7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f290c105d00 0x7f290c1997d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36308/0 (socket says 172.21.15.94:36308) 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.892+0000 7f290b7fe700 1 -- 172.21.15.94:0/1020945928 learned_addr learned my addr 172.21.15.94:0/1020945928 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.892+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651943512 0 0) 0x7f290c199ee0 con 0x7f290c105d00 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.892+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f8005530 con 0x7f290c105d00 2024-03-12T03:08:37.696 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.892+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3331219624 0 0) 0x7f28f8005530 con 0x7f290c105d00 2024-03-12T03:08:37.697 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.892+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f290c199ee0 con 0x7f290c105d00 2024-03-12T03:08:37.697 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.892+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f28fc001080 con 0x7f290c105d00 2024-03-12T03:08:37.697 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.892+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3571953203 0 0) 0x7f290c199ee0 con 0x7f290c105d00 2024-03-12T03:08:37.697 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.892+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c19a0b0 con 0x7f290c105d00 2024-03-12T03:08:37.697 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.892+0000 7f2912011700 1 -- 172.21.15.94:0/1020945928 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290c19a3a0 con 0x7f290c105d00 2024-03-12T03:08:37.697 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.892+0000 7f2912011700 1 -- 172.21.15.94:0/1020945928 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f290c19def0 con 0x7f290c105d00 2024-03-12T03:08:37.697 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.893+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f28fc001900 con 0x7f290c105d00 2024-03-12T03:08:37.697 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.893+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f28fc006000 con 0x7f290c105d00 2024-03-12T03:08:37.697 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.893+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 7) v1 ==== 45965+0+0 (unknown 4271366492 0 0) 0x7f28fc011620 con 0x7f290c105d00 2024-03-12T03:08:37.697 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.893+0000 7f2912011700 1 -- 172.21.15.94:0/1020945928 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28ec005150 con 0x7f290c105d00 2024-03-12T03:08:37.697 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.893+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 4125034817 0 0) 0x7f28fc047540 con 0x7f290c105d00 2024-03-12T03:08:37.697 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.898+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f28fc017560 con 0x7f290c105d00 2024-03-12T03:08:37.697 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:36.903+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mgrmap(e 8) v1 ==== 46071+0+0 (unknown 2158625433 0 0) 0x7f28fc0460b0 con 0x7f290c105d00 2024-03-12T03:08:37.698 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:37.110+0000 7f2912011700 1 -- 172.21.15.94:0/1020945928 --> v1:172.21.15.94:6800/3548280253 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f28ec000bc0 con 0x7f28f803a1d0 2024-03-12T03:08:37.698 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:37.114+0000 7f29097fa700 1 -- 172.21.15.94:0/1020945928 <== mgr.14120 v1:172.21.15.94:6800/3548280253 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7f28ec000bc0 con 0x7f28f803a1d0 2024-03-12T03:08:37.698 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:37.117+0000 7f2912011700 1 -- 172.21.15.94:0/1020945928 >> v1:172.21.15.94:6800/3548280253 conn(0x7f28f803a1d0 legacy=0x7f28f803c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:37.698 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:37.117+0000 7f2912011700 1 -- 172.21.15.94:0/1020945928 >> v1:172.21.15.94:6789/0 conn(0x7f290c105d00 legacy=0x7f290c1997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:37.698 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:37.117+0000 7f2912011700 1 -- 172.21.15.94:0/1020945928 shutdown_connections 2024-03-12T03:08:37.698 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:37.117+0000 7f2912011700 1 -- 172.21.15.94:0/1020945928 >> 172.21.15.94:0/1020945928 conn(0x7f290c1012a0 msgr2=0x7f290c1036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:37.698 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:37.117+0000 7f2912011700 1 -- 172.21.15.94:0/1020945928 shutdown_connections 2024-03-12T03:08:37.698 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:37.117+0000 7f2912011700 1 -- 172.21.15.94:0/1020945928 wait complete. 2024-03-12T03:08:37.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:37 smithi094 ceph-mon[29291]: [12/Mar/2024:03:08:36] ENGINE Bus STARTING 2024-03-12T03:08:37.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:37 smithi094 ceph-mon[29291]: [12/Mar/2024:03:08:36] ENGINE Serving on http://172.21.15.94:8765 2024-03-12T03:08:37.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:37 smithi094 ceph-mon[29291]: [12/Mar/2024:03:08:36] ENGINE Serving on https://172.21.15.94:7150 2024-03-12T03:08:37.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:37 smithi094 ceph-mon[29291]: [12/Mar/2024:03:08:36] ENGINE Bus STARTED 2024-03-12T03:08:37.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:37 smithi094 ceph-mon[29291]: mgrmap e8: y(active, since 2s) 2024-03-12T03:08:37.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:37 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:37.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:37 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:08:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:38 smithi094 ceph-mon[29291]: from='client.14132 v1:172.21.15.94:0/1020945928' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:08:38.976 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-12T03:08:38.976 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.163+0000 7fc17b2f3700 1 Processor -- start 2024-03-12T03:08:38.976 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.163+0000 7fc17b2f3700 1 -- start start 2024-03-12T03:08:38.977 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.163+0000 7fc17b2f3700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc17407a2e0 con 0x7fc17407b6e0 2024-03-12T03:08:38.977 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.163+0000 7fc17908f700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc17407b6e0 0x7fc174079b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36316/0 (socket says 172.21.15.94:36316) 2024-03-12T03:08:38.977 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.163+0000 7fc17908f700 1 -- 172.21.15.94:0/1551085499 learned_addr learned my addr 172.21.15.94:0/1551085499 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:38.977 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.164+0000 7fc16bfff700 1 -- 172.21.15.94:0/1551085499 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686132738 0 0) 0x7fc17407a2e0 con 0x7fc17407b6e0 2024-03-12T03:08:38.977 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.164+0000 7fc16bfff700 1 -- 172.21.15.94:0/1551085499 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1600035b0 con 0x7fc17407b6e0 2024-03-12T03:08:38.977 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.164+0000 7fc16bfff700 1 -- 172.21.15.94:0/1551085499 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 947146713 0 0) 0x7fc1600035b0 con 0x7fc17407b6e0 2024-03-12T03:08:38.977 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.164+0000 7fc16bfff700 1 -- 172.21.15.94:0/1551085499 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17407a4b0 con 0x7fc17407b6e0 2024-03-12T03:08:38.977 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.164+0000 7fc16bfff700 1 -- 172.21.15.94:0/1551085499 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fc170001580 con 0x7fc17407b6e0 2024-03-12T03:08:38.977 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.164+0000 7fc16bfff700 1 -- 172.21.15.94:0/1551085499 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7fc170001d20 con 0x7fc17407b6e0 2024-03-12T03:08:38.977 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.164+0000 7fc16bfff700 1 -- 172.21.15.94:0/1551085499 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fc170005960 con 0x7fc17407b6e0 2024-03-12T03:08:38.977 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.164+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1551085499 >> v1:172.21.15.94:6789/0 conn(0x7fc17407b6e0 legacy=0x7fc174079b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:38.977 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.164+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1551085499 shutdown_connections 2024-03-12T03:08:38.977 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.164+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1551085499 >> 172.21.15.94:0/1551085499 conn(0x7fc1741012a0 msgr2=0x7fc174103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:38.978 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.165+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1551085499 shutdown_connections 2024-03-12T03:08:38.978 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.165+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1551085499 wait complete. 2024-03-12T03:08:38.978 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.165+0000 7fc17b2f3700 1 Processor -- start 2024-03-12T03:08:38.978 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.165+0000 7fc17b2f3700 1 -- start start 2024-03-12T03:08:38.978 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.165+0000 7fc17b2f3700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc17419c110 con 0x7fc17407b6e0 2024-03-12T03:08:38.978 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.165+0000 7fc17908f700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc17407b6e0 0x7fc17419ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36320/0 (socket says 172.21.15.94:36320) 2024-03-12T03:08:38.978 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.165+0000 7fc17908f700 1 -- 172.21.15.94:0/1386124207 learned_addr learned my addr 172.21.15.94:0/1386124207 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:38.978 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.166+0000 7fc16a7fc700 1 -- 172.21.15.94:0/1386124207 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028020477 0 0) 0x7fc17419c110 con 0x7fc17407b6e0 2024-03-12T03:08:38.978 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.166+0000 7fc16a7fc700 1 -- 172.21.15.94:0/1386124207 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc160005530 con 0x7fc17407b6e0 2024-03-12T03:08:38.978 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.166+0000 7fc16a7fc700 1 -- 172.21.15.94:0/1386124207 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3116600735 0 0) 0x7fc160005530 con 0x7fc17407b6e0 2024-03-12T03:08:38.978 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.166+0000 7fc16a7fc700 1 -- 172.21.15.94:0/1386124207 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc17419c110 con 0x7fc17407b6e0 2024-03-12T03:08:38.978 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.166+0000 7fc16a7fc700 1 -- 172.21.15.94:0/1386124207 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fc170001080 con 0x7fc17407b6e0 2024-03-12T03:08:38.978 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.166+0000 7fc16a7fc700 1 -- 172.21.15.94:0/1386124207 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3742040069 0 0) 0x7fc17419c110 con 0x7fc17407b6e0 2024-03-12T03:08:38.979 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.166+0000 7fc16a7fc700 1 -- 172.21.15.94:0/1386124207 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17419c2e0 con 0x7fc17407b6e0 2024-03-12T03:08:38.979 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.166+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1386124207 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17419c5d0 con 0x7fc17407b6e0 2024-03-12T03:08:38.979 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.166+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1386124207 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1741a0120 con 0x7fc17407b6e0 2024-03-12T03:08:38.979 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.166+0000 7fc16a7fc700 1 -- 172.21.15.94:0/1386124207 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7fc170001cc0 con 0x7fc17407b6e0 2024-03-12T03:08:38.979 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.166+0000 7fc16a7fc700 1 -- 172.21.15.94:0/1386124207 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fc170006010 con 0x7fc17407b6e0 2024-03-12T03:08:38.979 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.167+0000 7fc16a7fc700 1 -- 172.21.15.94:0/1386124207 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 8) v1 ==== 46071+0+0 (unknown 2158625433 0 0) 0x7fc1700115b0 con 0x7fc17407b6e0 2024-03-12T03:08:38.979 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.167+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1386124207 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc158005150 con 0x7fc17407b6e0 2024-03-12T03:08:38.979 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.167+0000 7fc16a7fc700 1 -- 172.21.15.94:0/1386124207 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 4125034817 0 0) 0x7fc170047680 con 0x7fc17407b6e0 2024-03-12T03:08:38.979 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.172+0000 7fc16a7fc700 1 -- 172.21.15.94:0/1386124207 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fc1700174c0 con 0x7fc17407b6e0 2024-03-12T03:08:38.979 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.365+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1386124207 --> v1:172.21.15.94:6800/3548280253 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fc158000bc0 con 0x7fc16003a390 2024-03-12T03:08:38.979 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.366+0000 7fc16a7fc700 1 -- 172.21.15.94:0/1386124207 <== mgr.14120 v1:172.21.15.94:6800/3548280253 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (unknown 0 0 2070689548) 0x7fc158000bc0 con 0x7fc16003a390 2024-03-12T03:08:38.979 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.369+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1386124207 >> v1:172.21.15.94:6800/3548280253 conn(0x7fc16003a390 legacy=0x7fc16003c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:38.979 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.369+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1386124207 >> v1:172.21.15.94:6789/0 conn(0x7fc17407b6e0 legacy=0x7fc17419ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:38.980 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.369+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1386124207 shutdown_connections 2024-03-12T03:08:38.980 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.369+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1386124207 >> 172.21.15.94:0/1386124207 conn(0x7fc1741012a0 msgr2=0x7fc17410a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:38.980 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.369+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1386124207 shutdown_connections 2024-03-12T03:08:38.980 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:38.369+0000 7fc17b2f3700 1 -- 172.21.15.94:0/1386124207 wait complete. 2024-03-12T03:08:38.980 INFO:teuthology.orchestra.run.smithi094.stdout:Generating ssh key... 2024-03-12T03:08:39.909 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: Generating public/private rsa key pair. 2024-03-12T03:08:39.909 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: Your identification has been saved in /tmp/tmp3jx4r_3_/key. 2024-03-12T03:08:39.909 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: Your public key has been saved in /tmp/tmp3jx4r_3_/key.pub. 2024-03-12T03:08:39.909 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: The key fingerprint is: 2024-03-12T03:08:39.909 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: SHA256:Og1ubdlsmL5bwgZVQgtUte29B8HSmALnKaL/ILRKzT8 ceph-964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:08:39.909 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: The key's randomart image is: 2024-03-12T03:08:39.910 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: +---[RSA 3072]----+ 2024-03-12T03:08:39.910 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: | .oo=.+ | 2024-03-12T03:08:39.910 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: | . B + = | 2024-03-12T03:08:39.910 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: | . + = = + | 2024-03-12T03:08:39.910 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: | . o . o o . | 2024-03-12T03:08:39.910 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: | o o S . o | 2024-03-12T03:08:39.910 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: | + + B * o | 2024-03-12T03:08:39.910 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: | . = B @ = . .| 2024-03-12T03:08:39.910 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: | . . +EB + . | 2024-03-12T03:08:39.911 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: | . ..=o | 2024-03-12T03:08:39.911 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: +----[SHA256]-----+ 2024-03-12T03:08:40.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-mon[29291]: from='client.14134 v1:172.21.15.94:0/1386124207' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:08:40.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:40.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:39 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:40.416 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.423+0000 7f7130b6c700 1 Processor -- start 2024-03-12T03:08:40.416 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.424+0000 7f7130b6c700 1 -- start start 2024-03-12T03:08:40.416 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.424+0000 7f7130b6c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f712c07a2e0 con 0x7f712c07b6e0 2024-03-12T03:08:40.416 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.424+0000 7f712a59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f712c07b6e0 0x7f712c079b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36328/0 (socket says 172.21.15.94:36328) 2024-03-12T03:08:40.416 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.424+0000 7f712a59c700 1 -- 172.21.15.94:0/896164526 learned_addr learned my addr 172.21.15.94:0/896164526 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:40.416 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.424+0000 7f712959a700 1 -- 172.21.15.94:0/896164526 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486796654 0 0) 0x7f712c07a2e0 con 0x7f712c07b6e0 2024-03-12T03:08:40.416 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.424+0000 7f712959a700 1 -- 172.21.15.94:0/896164526 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71140035b0 con 0x7f712c07b6e0 2024-03-12T03:08:40.416 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.425+0000 7f712959a700 1 -- 172.21.15.94:0/896164526 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1622942308 0 0) 0x7f71140035b0 con 0x7f712c07b6e0 2024-03-12T03:08:40.416 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.425+0000 7f712959a700 1 -- 172.21.15.94:0/896164526 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c07a4b0 con 0x7f712c07b6e0 2024-03-12T03:08:40.416 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.425+0000 7f712959a700 1 -- 172.21.15.94:0/896164526 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f7120001580 con 0x7f712c07b6e0 2024-03-12T03:08:40.416 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.425+0000 7f712959a700 1 -- 172.21.15.94:0/896164526 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f7120001d20 con 0x7f712c07b6e0 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.425+0000 7f712959a700 1 -- 172.21.15.94:0/896164526 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f7120005960 con 0x7f712c07b6e0 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.425+0000 7f7130b6c700 1 -- 172.21.15.94:0/896164526 >> v1:172.21.15.94:6789/0 conn(0x7f712c07b6e0 legacy=0x7f712c079b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.425+0000 7f7130b6c700 1 -- 172.21.15.94:0/896164526 shutdown_connections 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.425+0000 7f7130b6c700 1 -- 172.21.15.94:0/896164526 >> 172.21.15.94:0/896164526 conn(0x7f712c101260 msgr2=0x7f712c1036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.425+0000 7f7130b6c700 1 -- 172.21.15.94:0/896164526 shutdown_connections 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.425+0000 7f7130b6c700 1 -- 172.21.15.94:0/896164526 wait complete. 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.426+0000 7f7130b6c700 1 Processor -- start 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.426+0000 7f7130b6c700 1 -- start start 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.426+0000 7f7130b6c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f712c19c110 con 0x7f712c07b6e0 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.426+0000 7f712a59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f712c07b6e0 0x7f712c19ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36338/0 (socket says 172.21.15.94:36338) 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.426+0000 7f712a59c700 1 -- 172.21.15.94:0/3452168738 learned_addr learned my addr 172.21.15.94:0/3452168738 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.427+0000 7f711b7fe700 1 -- 172.21.15.94:0/3452168738 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257888381 0 0) 0x7f712c19c110 con 0x7f712c07b6e0 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.427+0000 7f711b7fe700 1 -- 172.21.15.94:0/3452168738 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7114005530 con 0x7f712c07b6e0 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.427+0000 7f711b7fe700 1 -- 172.21.15.94:0/3452168738 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2732744632 0 0) 0x7f7114005530 con 0x7f712c07b6e0 2024-03-12T03:08:40.417 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.427+0000 7f711b7fe700 1 -- 172.21.15.94:0/3452168738 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f712c19c110 con 0x7f712c07b6e0 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.427+0000 7f711b7fe700 1 -- 172.21.15.94:0/3452168738 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f7120001080 con 0x7f712c07b6e0 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.427+0000 7f711b7fe700 1 -- 172.21.15.94:0/3452168738 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1928214456 0 0) 0x7f712c19c110 con 0x7f712c07b6e0 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.427+0000 7f711b7fe700 1 -- 172.21.15.94:0/3452168738 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c19c2e0 con 0x7f712c07b6e0 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.427+0000 7f7130b6c700 1 -- 172.21.15.94:0/3452168738 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712c19c5d0 con 0x7f712c07b6e0 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.427+0000 7f7130b6c700 1 -- 172.21.15.94:0/3452168738 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c1a0120 con 0x7f712c07b6e0 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.427+0000 7f711b7fe700 1 -- 172.21.15.94:0/3452168738 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f7120001c00 con 0x7f712c07b6e0 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.427+0000 7f711b7fe700 1 -- 172.21.15.94:0/3452168738 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f7120005f30 con 0x7f712c07b6e0 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.428+0000 7f711b7fe700 1 -- 172.21.15.94:0/3452168738 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 8) v1 ==== 46071+0+0 (unknown 2158625433 0 0) 0x7f71200115f0 con 0x7f712c07b6e0 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.428+0000 7f7130b6c700 1 -- 172.21.15.94:0/3452168738 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710c005150 con 0x7f712c07b6e0 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.428+0000 7f711b7fe700 1 -- 172.21.15.94:0/3452168738 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 4125034817 0 0) 0x7f7120047840 con 0x7f712c07b6e0 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.433+0000 7f711b7fe700 1 -- 172.21.15.94:0/3452168738 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f7120017540 con 0x7f712c07b6e0 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.627+0000 7f7130b6c700 1 -- 172.21.15.94:0/3452168738 --> v1:172.21.15.94:6800/3548280253 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f710c000bc0 con 0x7f711403a340 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.771+0000 7f711b7fe700 1 -- 172.21.15.94:0/3452168738 <== mgr.14120 v1:172.21.15.94:6800/3548280253 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7f710c000bc0 con 0x7f711403a340 2024-03-12T03:08:40.418 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.773+0000 7f7130b6c700 1 -- 172.21.15.94:0/3452168738 >> v1:172.21.15.94:6800/3548280253 conn(0x7f711403a340 legacy=0x7f711403c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:40.419 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.773+0000 7f7130b6c700 1 -- 172.21.15.94:0/3452168738 >> v1:172.21.15.94:6789/0 conn(0x7f712c07b6e0 legacy=0x7f712c19ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:40.419 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.773+0000 7f7130b6c700 1 -- 172.21.15.94:0/3452168738 shutdown_connections 2024-03-12T03:08:40.419 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.773+0000 7f7130b6c700 1 -- 172.21.15.94:0/3452168738 >> 172.21.15.94:0/3452168738 conn(0x7f712c101260 msgr2=0x7f712c10a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:40.419 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.774+0000 7f7130b6c700 1 -- 172.21.15.94:0/3452168738 shutdown_connections 2024-03-12T03:08:40.419 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:39.774+0000 7f7130b6c700 1 -- 172.21.15.94:0/3452168738 wait complete. 2024-03-12T03:08:41.063 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:40 smithi094 ceph-mon[29291]: from='client.14136 v1:172.21.15.94:0/3452168738' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:08:41.063 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:40 smithi094 ceph-mon[29291]: Generating ssh key... 2024-03-12T03:08:41.681 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiwCWw9OXTQjngODjhbgtVKcfxbxEg/jlDJd4myYgBMFKsXT4DYeh0rxng9JViU9MlZJbgFOil0gNMEvcS3bVJPJFkGYdOeaXcAWT6Z2r7ohukEib5AY/Pr9+SpMBwyh+pEmt/2npEC3Nap300wGy3krPs4uoFv8MZbOuvjIW2h5CgxMM7k9jiArk4B69fzzk9E1r/jCxQKAXuRJsWN8fR5BLF13g97cQJmuotj8kGRrDykwnW9X7aBOV7AFuOvcCP4JSSyzydVrMHAFcGEOYoA1Az3MV+xwAKuxcxd41xvoQXaJOqRIGSPJCDfPWPGgiYmG38r/lTCYAtGMmXJbhfflny1spaeCefNnsIVU43Z8abyjDyK6xW1anZaqD2uVbortx+A6NYmn6caW1ak8VQUgvtb6VjtEYjw3Lbp0DtVFbvS2mJp3nj/k4GGSdkIMLH01LfMKAi8oFoviGqCUsX9FdNqxxr1HubRwbOR20UQQrzVtAIx50Oc4Zwie2Ma60= ceph-964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:08:41.682 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.817+0000 7f8475ff4700 1 Processor -- start 2024-03-12T03:08:41.682 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.817+0000 7f8475ff4700 1 -- start start 2024-03-12T03:08:41.682 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.817+0000 7f8475ff4700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f84701090a0 con 0x7f8470106290 2024-03-12T03:08:41.682 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.817+0000 7f846f7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8470106290 0x7f84701066b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36348/0 (socket says 172.21.15.94:36348) 2024-03-12T03:08:41.682 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.817+0000 7f846f7fe700 1 -- 172.21.15.94:0/3749032056 learned_addr learned my addr 172.21.15.94:0/3749032056 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:41.682 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.817+0000 7f846e7fc700 1 -- 172.21.15.94:0/3749032056 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671315886 0 0) 0x7f84701090a0 con 0x7f8470106290 2024-03-12T03:08:41.682 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.817+0000 7f846e7fc700 1 -- 172.21.15.94:0/3749032056 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84580035b0 con 0x7f8470106290 2024-03-12T03:08:41.682 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.817+0000 7f846e7fc700 1 -- 172.21.15.94:0/3749032056 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 620621077 0 0) 0x7f84580035b0 con 0x7f8470106290 2024-03-12T03:08:41.682 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.817+0000 7f846e7fc700 1 -- 172.21.15.94:0/3749032056 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847010a280 con 0x7f8470106290 2024-03-12T03:08:41.682 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.817+0000 7f846e7fc700 1 -- 172.21.15.94:0/3749032056 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f8460001580 con 0x7f8470106290 2024-03-12T03:08:41.682 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.818+0000 7f846e7fc700 1 -- 172.21.15.94:0/3749032056 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f8460001d20 con 0x7f8470106290 2024-03-12T03:08:41.682 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.818+0000 7f846e7fc700 1 -- 172.21.15.94:0/3749032056 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f8460005960 con 0x7f8470106290 2024-03-12T03:08:41.682 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.818+0000 7f8475ff4700 1 -- 172.21.15.94:0/3749032056 >> v1:172.21.15.94:6789/0 conn(0x7f8470106290 legacy=0x7f84701066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.818+0000 7f8475ff4700 1 -- 172.21.15.94:0/3749032056 shutdown_connections 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.818+0000 7f8475ff4700 1 -- 172.21.15.94:0/3749032056 >> 172.21.15.94:0/3749032056 conn(0x7f84700ff8f0 msgr2=0x7f8470101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.818+0000 7f8475ff4700 1 -- 172.21.15.94:0/3749032056 shutdown_connections 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.818+0000 7f8475ff4700 1 -- 172.21.15.94:0/3749032056 wait complete. 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.819+0000 7f8475ff4700 1 Processor -- start 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.819+0000 7f8475ff4700 1 -- start start 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.819+0000 7f8475ff4700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8470199f50 con 0x7f8470106290 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.819+0000 7f846f7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8470106290 0x7f8470199840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36364/0 (socket says 172.21.15.94:36364) 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.819+0000 7f846f7fe700 1 -- 172.21.15.94:0/2704343385 learned_addr learned my addr 172.21.15.94:0/2704343385 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.819+0000 7f846cff9700 1 -- 172.21.15.94:0/2704343385 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833559748 0 0) 0x7f8470199f50 con 0x7f8470106290 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.819+0000 7f846cff9700 1 -- 172.21.15.94:0/2704343385 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8458005530 con 0x7f8470106290 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.820+0000 7f846cff9700 1 -- 172.21.15.94:0/2704343385 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2818994556 0 0) 0x7f8458005530 con 0x7f8470106290 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.820+0000 7f846cff9700 1 -- 172.21.15.94:0/2704343385 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8470199f50 con 0x7f8470106290 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.820+0000 7f846cff9700 1 -- 172.21.15.94:0/2704343385 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f8460004870 con 0x7f8470106290 2024-03-12T03:08:41.683 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.820+0000 7f846cff9700 1 -- 172.21.15.94:0/2704343385 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3628853038 0 0) 0x7f8470199f50 con 0x7f8470106290 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.820+0000 7f846cff9700 1 -- 172.21.15.94:0/2704343385 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847019a120 con 0x7f8470106290 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.820+0000 7f8475ff4700 1 -- 172.21.15.94:0/2704343385 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847019a410 con 0x7f8470106290 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.820+0000 7f8475ff4700 1 -- 172.21.15.94:0/2704343385 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f847019df60 con 0x7f8470106290 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.820+0000 7f846cff9700 1 -- 172.21.15.94:0/2704343385 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f8460001d20 con 0x7f8470106290 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.820+0000 7f846cff9700 1 -- 172.21.15.94:0/2704343385 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f8460006010 con 0x7f8470106290 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.821+0000 7f846cff9700 1 -- 172.21.15.94:0/2704343385 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 8) v1 ==== 46071+0+0 (unknown 2158625433 0 0) 0x7f84600115f0 con 0x7f8470106290 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.821+0000 7f8475ff4700 1 -- 172.21.15.94:0/2704343385 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845c005120 con 0x7f8470106290 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.821+0000 7f846cff9700 1 -- 172.21.15.94:0/2704343385 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 4125034817 0 0) 0x7f8460047710 con 0x7f8470106290 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:40.826+0000 7f846cff9700 1 -- 172.21.15.94:0/2704343385 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f8460017590 con 0x7f8470106290 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:41.022+0000 7f8475ff4700 1 -- 172.21.15.94:0/2704343385 --> v1:172.21.15.94:6800/3548280253 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f845c000b90 con 0x7f845803a1a0 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:41.023+0000 7f846cff9700 1 -- 172.21.15.94:0/2704343385 <== mgr.14120 v1:172.21.15.94:6800/3548280253 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (unknown 0 0 2104041102) 0x7f845c000b90 con 0x7f845803a1a0 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:41.025+0000 7f8475ff4700 1 -- 172.21.15.94:0/2704343385 >> v1:172.21.15.94:6800/3548280253 conn(0x7f845803a1a0 legacy=0x7f845803c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:41.025+0000 7f8475ff4700 1 -- 172.21.15.94:0/2704343385 >> v1:172.21.15.94:6789/0 conn(0x7f8470106290 legacy=0x7f8470199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:41.025+0000 7f8475ff4700 1 -- 172.21.15.94:0/2704343385 shutdown_connections 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:41.025+0000 7f8475ff4700 1 -- 172.21.15.94:0/2704343385 >> 172.21.15.94:0/2704343385 conn(0x7f84700ff8f0 msgr2=0x7f8470103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:41.684 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:41.025+0000 7f8475ff4700 1 -- 172.21.15.94:0/2704343385 shutdown_connections 2024-03-12T03:08:41.685 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:41.025+0000 7f8475ff4700 1 -- 172.21.15.94:0/2704343385 wait complete. 2024-03-12T03:08:41.685 INFO:teuthology.orchestra.run.smithi094.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-12T03:08:41.685 INFO:teuthology.orchestra.run.smithi094.stdout:Adding key to root@localhost authorized_keys... 2024-03-12T03:08:41.685 INFO:teuthology.orchestra.run.smithi094.stdout:Adding host smithi094... 2024-03-12T03:08:41.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:41 smithi094 ceph-mon[29291]: from='client.14138 v1:172.21.15.94:0/2704343385' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:08:43.184 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:43 smithi094 ceph-mon[29291]: from='client.14140 v1:172.21.15.94:0/3014763428' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi094", "addr": "172.21.15.94", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:08:43.184 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:43 smithi094 ceph-mon[29291]: Deploying cephadm binary to smithi094 2024-03-12T03:08:47.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout Added host 'smithi094' with addr '172.21.15.94' 2024-03-12T03:08:47.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.135+0000 7f188ef09700 1 Processor -- start 2024-03-12T03:08:47.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.135+0000 7f188ef09700 1 -- start start 2024-03-12T03:08:47.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.135+0000 7f188ef09700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1888079690 con 0x7f18881067c0 2024-03-12T03:08:47.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.136+0000 7f188cca5700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f18881067c0 0x7f1888108bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36368/0 (socket says 172.21.15.94:36368) 2024-03-12T03:08:47.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.136+0000 7f188cca5700 1 -- 172.21.15.94:0/2797168475 learned_addr learned my addr 172.21.15.94:0/2797168475 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:47.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.136+0000 7f18877fe700 1 -- 172.21.15.94:0/2797168475 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492449886 0 0) 0x7f1888079690 con 0x7f18881067c0 2024-03-12T03:08:47.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.136+0000 7f18877fe700 1 -- 172.21.15.94:0/2797168475 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18780035b0 con 0x7f18881067c0 2024-03-12T03:08:47.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.136+0000 7f18877fe700 1 -- 172.21.15.94:0/2797168475 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3403262082 0 0) 0x7f18780035b0 con 0x7f18881067c0 2024-03-12T03:08:47.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.136+0000 7f18877fe700 1 -- 172.21.15.94:0/2797168475 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188810b2e0 con 0x7f18881067c0 2024-03-12T03:08:47.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.136+0000 7f18877fe700 1 -- 172.21.15.94:0/2797168475 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f187c001580 con 0x7f18881067c0 2024-03-12T03:08:47.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.136+0000 7f18877fe700 1 -- 172.21.15.94:0/2797168475 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f187c001d20 con 0x7f18881067c0 2024-03-12T03:08:47.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.136+0000 7f18877fe700 1 -- 172.21.15.94:0/2797168475 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f187c005960 con 0x7f18881067c0 2024-03-12T03:08:47.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.137+0000 7f188ef09700 1 -- 172.21.15.94:0/2797168475 >> v1:172.21.15.94:6789/0 conn(0x7f18881067c0 legacy=0x7f1888108bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:47.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.137+0000 7f188ef09700 1 -- 172.21.15.94:0/2797168475 shutdown_connections 2024-03-12T03:08:47.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.137+0000 7f188ef09700 1 -- 172.21.15.94:0/2797168475 >> 172.21.15.94:0/2797168475 conn(0x7f18881001b0 msgr2=0x7f18881025f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:47.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.137+0000 7f188ef09700 1 -- 172.21.15.94:0/2797168475 shutdown_connections 2024-03-12T03:08:47.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.137+0000 7f188ef09700 1 -- 172.21.15.94:0/2797168475 wait complete. 2024-03-12T03:08:47.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.138+0000 7f188ef09700 1 Processor -- start 2024-03-12T03:08:47.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.138+0000 7f188ef09700 1 -- start start 2024-03-12T03:08:47.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.138+0000 7f188ef09700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1888199f00 con 0x7f18881067c0 2024-03-12T03:08:47.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.138+0000 7f188cca5700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f18881067c0 0x7f18881997f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36374/0 (socket says 172.21.15.94:36374) 2024-03-12T03:08:47.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.138+0000 7f188cca5700 1 -- 172.21.15.94:0/3014763428 learned_addr learned my addr 172.21.15.94:0/3014763428 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:47.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.138+0000 7f1885ffb700 1 -- 172.21.15.94:0/3014763428 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215732148 0 0) 0x7f1888199f00 con 0x7f18881067c0 2024-03-12T03:08:47.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.138+0000 7f1885ffb700 1 -- 172.21.15.94:0/3014763428 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1878005530 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.138+0000 7f1885ffb700 1 -- 172.21.15.94:0/3014763428 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2516010520 0 0) 0x7f1878005530 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.138+0000 7f1885ffb700 1 -- 172.21.15.94:0/3014763428 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1888199f00 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.138+0000 7f1885ffb700 1 -- 172.21.15.94:0/3014763428 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f187c004870 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.138+0000 7f1885ffb700 1 -- 172.21.15.94:0/3014763428 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3599439662 0 0) 0x7f1888199f00 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.138+0000 7f1885ffb700 1 -- 172.21.15.94:0/3014763428 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188819a0d0 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.139+0000 7f188ef09700 1 -- 172.21.15.94:0/3014763428 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f188819a3c0 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.139+0000 7f1885ffb700 1 -- 172.21.15.94:0/3014763428 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f187c004ab0 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.139+0000 7f1885ffb700 1 -- 172.21.15.94:0/3014763428 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f187c006390 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.139+0000 7f188ef09700 1 -- 172.21.15.94:0/3014763428 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f188819df10 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.139+0000 7f1885ffb700 1 -- 172.21.15.94:0/3014763428 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 8) v1 ==== 46071+0+0 (unknown 2158625433 0 0) 0x7f187c011a80 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.140+0000 7f1885ffb700 1 -- 172.21.15.94:0/3014763428 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 4125034817 0 0) 0x7f187c048940 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.140+0000 7f188ef09700 1 -- 172.21.15.94:0/3014763428 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1888063410 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.145+0000 7f1885ffb700 1 -- 172.21.15.94:0/3014763428 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f187c0178d0 con 0x7f18881067c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:42.350+0000 7f188ef09700 1 -- 172.21.15.94:0/3014763428 --> v1:172.21.15.94:6800/3548280253 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi094", "addr": "172.21.15.94", "target": ["mon-mgr", ""]}) v1 -- 0x7f188802d940 con 0x7f187803a1c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:46.553+0000 7f1885ffb700 1 -- 172.21.15.94:0/3014763428 <== mgr.14120 v1:172.21.15.94:6800/3548280253 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (unknown 0 0 3568569481) 0x7f188802d940 con 0x7f187803a1c0 2024-03-12T03:08:47.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:46.556+0000 7f188ef09700 1 -- 172.21.15.94:0/3014763428 >> v1:172.21.15.94:6800/3548280253 conn(0x7f187803a1c0 legacy=0x7f187803c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:47.184 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:46.556+0000 7f188ef09700 1 -- 172.21.15.94:0/3014763428 >> v1:172.21.15.94:6789/0 conn(0x7f18881067c0 legacy=0x7f18881997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:47.184 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:46.556+0000 7f188ef09700 1 -- 172.21.15.94:0/3014763428 shutdown_connections 2024-03-12T03:08:47.184 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:46.556+0000 7f188ef09700 1 -- 172.21.15.94:0/3014763428 >> 172.21.15.94:0/3014763428 conn(0x7f18881001b0 msgr2=0x7f1888190470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:47.184 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:46.556+0000 7f188ef09700 1 -- 172.21.15.94:0/3014763428 shutdown_connections 2024-03-12T03:08:47.184 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:46.556+0000 7f188ef09700 1 -- 172.21.15.94:0/3014763428 wait complete. 2024-03-12T03:08:47.184 INFO:teuthology.orchestra.run.smithi094.stdout:Deploying unmanaged mon service... 2024-03-12T03:08:47.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:47 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:47.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:47 smithi094 ceph-mon[29291]: Added host smithi094 2024-03-12T03:08:47.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:47 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:08:49.128 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:48 smithi094 ceph-mon[29291]: from='client.14142 v1:172.21.15.94:0/3305933339' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:08:49.128 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:48 smithi094 ceph-mon[29291]: Saving service mon spec with placement count:5 2024-03-12T03:08:49.128 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:48 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:49.595 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-12T03:08:49.595 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.625+0000 7f363eaad700 1 Processor -- start 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.625+0000 7f363eaad700 1 -- start start 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.626+0000 7f363eaad700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f363810a130 con 0x7f3638105d00 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.626+0000 7f363c849700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3638105d00 0x7f3638106120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56112/0 (socket says 172.21.15.94:56112) 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.626+0000 7f363c849700 1 -- 172.21.15.94:0/730664625 learned_addr learned my addr 172.21.15.94:0/730664625 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.626+0000 7f36377fe700 1 -- 172.21.15.94:0/730664625 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275559479 0 0) 0x7f363810a130 con 0x7f3638105d00 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.626+0000 7f36377fe700 1 -- 172.21.15.94:0/730664625 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36280035b0 con 0x7f3638105d00 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.626+0000 7f36377fe700 1 -- 172.21.15.94:0/730664625 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3496385047 0 0) 0x7f36280035b0 con 0x7f3638105d00 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.626+0000 7f36377fe700 1 -- 172.21.15.94:0/730664625 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363810b310 con 0x7f3638105d00 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.626+0000 7f36377fe700 1 -- 172.21.15.94:0/730664625 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f362c001580 con 0x7f3638105d00 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.626+0000 7f36377fe700 1 -- 172.21.15.94:0/730664625 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f362c001c50 con 0x7f3638105d00 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.626+0000 7f36377fe700 1 -- 172.21.15.94:0/730664625 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f362c005960 con 0x7f3638105d00 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.627+0000 7f363eaad700 1 -- 172.21.15.94:0/730664625 >> v1:172.21.15.94:6789/0 conn(0x7f3638105d00 legacy=0x7f3638106120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.627+0000 7f363eaad700 1 -- 172.21.15.94:0/730664625 shutdown_connections 2024-03-12T03:08:49.596 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.627+0000 7f363eaad700 1 -- 172.21.15.94:0/730664625 >> 172.21.15.94:0/730664625 conn(0x7f36381012a0 msgr2=0x7f36381036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:49.597 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.627+0000 7f363eaad700 1 -- 172.21.15.94:0/730664625 shutdown_connections 2024-03-12T03:08:49.597 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.627+0000 7f363eaad700 1 -- 172.21.15.94:0/730664625 wait complete. 2024-03-12T03:08:49.597 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.627+0000 7f363eaad700 1 Processor -- start 2024-03-12T03:08:49.597 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.627+0000 7f363eaad700 1 -- start start 2024-03-12T03:08:49.597 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.627+0000 7f363eaad700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3638199f40 con 0x7f3638105d00 2024-03-12T03:08:49.597 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.628+0000 7f363c849700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3638105d00 0x7f3638199830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56128/0 (socket says 172.21.15.94:56128) 2024-03-12T03:08:49.597 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.628+0000 7f363c849700 1 -- 172.21.15.94:0/3305933339 learned_addr learned my addr 172.21.15.94:0/3305933339 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:49.597 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.628+0000 7f3635ffb700 1 -- 172.21.15.94:0/3305933339 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410246715 0 0) 0x7f3638199f40 con 0x7f3638105d00 2024-03-12T03:08:49.597 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.628+0000 7f3635ffb700 1 -- 172.21.15.94:0/3305933339 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3628005580 con 0x7f3638105d00 2024-03-12T03:08:49.597 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.628+0000 7f3635ffb700 1 -- 172.21.15.94:0/3305933339 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2820201090 0 0) 0x7f3628005580 con 0x7f3638105d00 2024-03-12T03:08:49.597 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.628+0000 7f3635ffb700 1 -- 172.21.15.94:0/3305933339 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3638199f40 con 0x7f3638105d00 2024-03-12T03:08:49.597 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.628+0000 7f3635ffb700 1 -- 172.21.15.94:0/3305933339 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f362c001080 con 0x7f3638105d00 2024-03-12T03:08:49.597 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.628+0000 7f3635ffb700 1 -- 172.21.15.94:0/3305933339 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2071408811 0 0) 0x7f3638199f40 con 0x7f3638105d00 2024-03-12T03:08:49.598 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.628+0000 7f3635ffb700 1 -- 172.21.15.94:0/3305933339 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363819a110 con 0x7f3638105d00 2024-03-12T03:08:49.598 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.628+0000 7f363eaad700 1 -- 172.21.15.94:0/3305933339 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f363819a400 con 0x7f3638105d00 2024-03-12T03:08:49.598 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.628+0000 7f3635ffb700 1 -- 172.21.15.94:0/3305933339 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3210529559 0 0) 0x7f362c005910 con 0x7f3638105d00 2024-03-12T03:08:49.598 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.628+0000 7f3635ffb700 1 -- 172.21.15.94:0/3305933339 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f362c006230 con 0x7f3638105d00 2024-03-12T03:08:49.598 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.628+0000 7f363eaad700 1 -- 172.21.15.94:0/3305933339 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f363819df50 con 0x7f3638105d00 2024-03-12T03:08:49.598 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.629+0000 7f3635ffb700 1 -- 172.21.15.94:0/3305933339 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 8) v1 ==== 46071+0+0 (unknown 2158625433 0 0) 0x7f362c011940 con 0x7f3638105d00 2024-03-12T03:08:49.598 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.629+0000 7f3635ffb700 1 -- 172.21.15.94:0/3305933339 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 4125034817 0 0) 0x7f362c047730 con 0x7f3638105d00 2024-03-12T03:08:49.598 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.629+0000 7f363eaad700 1 -- 172.21.15.94:0/3305933339 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3638063410 con 0x7f3638105d00 2024-03-12T03:08:49.598 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.634+0000 7f3635ffb700 1 -- 172.21.15.94:0/3305933339 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f362c0174d0 con 0x7f3638105d00 2024-03-12T03:08:49.598 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.843+0000 7f363eaad700 1 -- 172.21.15.94:0/3305933339 --> v1:172.21.15.94:6800/3548280253 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3638191530 con 0x7f362803a330 2024-03-12T03:08:49.598 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.848+0000 7f3635ffb700 1 -- 172.21.15.94:0/3305933339 <== mgr.14120 v1:172.21.15.94:6800/3548280253 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 3265049985) 0x7f3638191530 con 0x7f362803a330 2024-03-12T03:08:49.598 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.850+0000 7f363eaad700 1 -- 172.21.15.94:0/3305933339 >> v1:172.21.15.94:6800/3548280253 conn(0x7f362803a330 legacy=0x7f362803c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:49.598 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.850+0000 7f363eaad700 1 -- 172.21.15.94:0/3305933339 >> v1:172.21.15.94:6789/0 conn(0x7f3638105d00 legacy=0x7f3638199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:49.599 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.850+0000 7f363eaad700 1 -- 172.21.15.94:0/3305933339 shutdown_connections 2024-03-12T03:08:49.599 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.850+0000 7f363eaad700 1 -- 172.21.15.94:0/3305933339 >> 172.21.15.94:0/3305933339 conn(0x7f36381012a0 msgr2=0x7f36381036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:49.599 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.850+0000 7f363eaad700 1 -- 172.21.15.94:0/3305933339 shutdown_connections 2024-03-12T03:08:49.599 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:47.850+0000 7f363eaad700 1 -- 172.21.15.94:0/3305933339 wait complete. 2024-03-12T03:08:49.599 INFO:teuthology.orchestra.run.smithi094.stdout:Deploying unmanaged mgr service... 2024-03-12T03:08:50.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:50 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:50.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:50 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:51.376 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:51 smithi094 ceph-mon[29291]: from='client.14144 v1:172.21.15.94:0/2995082520' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:08:51.376 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:51 smithi094 ceph-mon[29291]: Saving service mgr spec with placement count:2 2024-03-12T03:08:51.376 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:51 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:51.376 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:51 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:51.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-12T03:08:51.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.091+0000 7fb725cc1700 1 Processor -- start 2024-03-12T03:08:51.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.091+0000 7fb725cc1700 1 -- start start 2024-03-12T03:08:51.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.091+0000 7fb725cc1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb7200796d0 con 0x7fb720106780 2024-03-12T03:08:51.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.091+0000 7fb71f7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb720106780 0x7fb720108b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56134/0 (socket says 172.21.15.94:56134) 2024-03-12T03:08:51.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.091+0000 7fb71f7fe700 1 -- 172.21.15.94:0/1017411936 learned_addr learned my addr 172.21.15.94:0/1017411936 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:51.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.091+0000 7fb71e7fc700 1 -- 172.21.15.94:0/1017411936 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009204296 0 0) 0x7fb7200796d0 con 0x7fb720106780 2024-03-12T03:08:51.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.091+0000 7fb71e7fc700 1 -- 172.21.15.94:0/1017411936 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7080035b0 con 0x7fb720106780 2024-03-12T03:08:51.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.092+0000 7fb71e7fc700 1 -- 172.21.15.94:0/1017411936 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 583872732 0 0) 0x7fb7080035b0 con 0x7fb720106780 2024-03-12T03:08:51.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.092+0000 7fb71e7fc700 1 -- 172.21.15.94:0/1017411936 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72010b2a0 con 0x7fb720106780 2024-03-12T03:08:51.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.092+0000 7fb71e7fc700 1 -- 172.21.15.94:0/1017411936 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fb710001580 con 0x7fb720106780 2024-03-12T03:08:51.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.092+0000 7fb71e7fc700 1 -- 172.21.15.94:0/1017411936 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fb710004930 con 0x7fb720106780 2024-03-12T03:08:51.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.092+0000 7fb71e7fc700 1 -- 172.21.15.94:0/1017411936 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fb710005a70 con 0x7fb720106780 2024-03-12T03:08:51.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.092+0000 7fb725cc1700 1 -- 172.21.15.94:0/1017411936 >> v1:172.21.15.94:6789/0 conn(0x7fb720106780 legacy=0x7fb720108b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:51.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.092+0000 7fb725cc1700 1 -- 172.21.15.94:0/1017411936 shutdown_connections 2024-03-12T03:08:51.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.092+0000 7fb725cc1700 1 -- 172.21.15.94:0/1017411936 >> 172.21.15.94:0/1017411936 conn(0x7fb7201001b0 msgr2=0x7fb7201025f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:51.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.092+0000 7fb725cc1700 1 -- 172.21.15.94:0/1017411936 shutdown_connections 2024-03-12T03:08:51.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.092+0000 7fb725cc1700 1 -- 172.21.15.94:0/1017411936 wait complete. 2024-03-12T03:08:51.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.093+0000 7fb725cc1700 1 Processor -- start 2024-03-12T03:08:51.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.093+0000 7fb725cc1700 1 -- start start 2024-03-12T03:08:51.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.093+0000 7fb725cc1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb720199f70 con 0x7fb720106780 2024-03-12T03:08:51.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.093+0000 7fb71f7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb720106780 0x7fb720199860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56136/0 (socket says 172.21.15.94:56136) 2024-03-12T03:08:51.387 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.093+0000 7fb71f7fe700 1 -- 172.21.15.94:0/2995082520 learned_addr learned my addr 172.21.15.94:0/2995082520 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:51.387 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.094+0000 7fb71cff9700 1 -- 172.21.15.94:0/2995082520 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365914938 0 0) 0x7fb720199f70 con 0x7fb720106780 2024-03-12T03:08:51.387 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.094+0000 7fb71cff9700 1 -- 172.21.15.94:0/2995082520 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb708005530 con 0x7fb720106780 2024-03-12T03:08:51.387 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.094+0000 7fb71cff9700 1 -- 172.21.15.94:0/2995082520 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2904183294 0 0) 0x7fb708005530 con 0x7fb720106780 2024-03-12T03:08:51.387 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.094+0000 7fb71cff9700 1 -- 172.21.15.94:0/2995082520 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb720199f70 con 0x7fb720106780 2024-03-12T03:08:51.387 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.094+0000 7fb71cff9700 1 -- 172.21.15.94:0/2995082520 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fb710001080 con 0x7fb720106780 2024-03-12T03:08:51.387 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.094+0000 7fb71cff9700 1 -- 172.21.15.94:0/2995082520 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 901566406 0 0) 0x7fb720199f70 con 0x7fb720106780 2024-03-12T03:08:51.387 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.094+0000 7fb71cff9700 1 -- 172.21.15.94:0/2995082520 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72019a140 con 0x7fb720106780 2024-03-12T03:08:51.387 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.094+0000 7fb725cc1700 1 -- 172.21.15.94:0/2995082520 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72019a430 con 0x7fb720106780 2024-03-12T03:08:51.387 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.094+0000 7fb725cc1700 1 -- 172.21.15.94:0/2995082520 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72019df80 con 0x7fb720106780 2024-03-12T03:08:51.387 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.094+0000 7fb71cff9700 1 -- 172.21.15.94:0/2995082520 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fb710004930 con 0x7fb720106780 2024-03-12T03:08:51.387 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.094+0000 7fb71cff9700 1 -- 172.21.15.94:0/2995082520 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fb710006050 con 0x7fb720106780 2024-03-12T03:08:51.387 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.095+0000 7fb71cff9700 1 -- 172.21.15.94:0/2995082520 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 8) v1 ==== 46071+0+0 (unknown 2158625433 0 0) 0x7fb710011610 con 0x7fb720106780 2024-03-12T03:08:51.388 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.095+0000 7fb71cff9700 1 -- 172.21.15.94:0/2995082520 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 4125034817 0 0) 0x7fb710047760 con 0x7fb720106780 2024-03-12T03:08:51.388 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.095+0000 7fb725cc1700 1 -- 172.21.15.94:0/2995082520 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70c005150 con 0x7fb720106780 2024-03-12T03:08:51.388 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.100+0000 7fb71cff9700 1 -- 172.21.15.94:0/2995082520 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fb710017560 con 0x7fb720106780 2024-03-12T03:08:51.388 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.340+0000 7fb725cc1700 1 -- 172.21.15.94:0/2995082520 --> v1:172.21.15.94:6800/3548280253 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb70c000bc0 con 0x7fb70803a410 2024-03-12T03:08:51.388 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.345+0000 7fb71cff9700 1 -- 172.21.15.94:0/2995082520 <== mgr.14120 v1:172.21.15.94:6800/3548280253 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 325935098) 0x7fb70c000bc0 con 0x7fb70803a410 2024-03-12T03:08:51.388 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.347+0000 7fb725cc1700 1 -- 172.21.15.94:0/2995082520 >> v1:172.21.15.94:6800/3548280253 conn(0x7fb70803a410 legacy=0x7fb70803c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:51.388 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.347+0000 7fb725cc1700 1 -- 172.21.15.94:0/2995082520 >> v1:172.21.15.94:6789/0 conn(0x7fb720106780 legacy=0x7fb720199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:51.388 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.347+0000 7fb725cc1700 1 -- 172.21.15.94:0/2995082520 shutdown_connections 2024-03-12T03:08:51.388 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.347+0000 7fb725cc1700 1 -- 172.21.15.94:0/2995082520 >> 172.21.15.94:0/2995082520 conn(0x7fb7201001b0 msgr2=0x7fb720190520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:51.388 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.347+0000 7fb725cc1700 1 -- 172.21.15.94:0/2995082520 shutdown_connections 2024-03-12T03:08:51.388 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:50.347+0000 7fb725cc1700 1 -- 172.21.15.94:0/2995082520 wait complete. 2024-03-12T03:08:52.714 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.847+0000 7fb6aaf2f700 1 Processor -- start 2024-03-12T03:08:52.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.847+0000 7fb6aaf2f700 1 -- start start 2024-03-12T03:08:52.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.847+0000 7fb6aaf2f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6a4109050 con 0x7fb6a41061b0 2024-03-12T03:08:52.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.848+0000 7fb6a8ccb700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb6a41061b0 0x7fb6a41065d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56138/0 (socket says 172.21.15.94:56138) 2024-03-12T03:08:52.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.848+0000 7fb6a8ccb700 1 -- 172.21.15.94:0/4230820852 learned_addr learned my addr 172.21.15.94:0/4230820852 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:52.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.848+0000 7fb6a37fe700 1 -- 172.21.15.94:0/4230820852 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836072981 0 0) 0x7fb6a4109050 con 0x7fb6a41061b0 2024-03-12T03:08:52.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.848+0000 7fb6a37fe700 1 -- 172.21.15.94:0/4230820852 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6940035b0 con 0x7fb6a41061b0 2024-03-12T03:08:52.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.848+0000 7fb6a37fe700 1 -- 172.21.15.94:0/4230820852 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3231613930 0 0) 0x7fb6940035b0 con 0x7fb6a41061b0 2024-03-12T03:08:52.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.848+0000 7fb6a37fe700 1 -- 172.21.15.94:0/4230820852 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a410a230 con 0x7fb6a41061b0 2024-03-12T03:08:52.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.848+0000 7fb6a37fe700 1 -- 172.21.15.94:0/4230820852 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fb698001580 con 0x7fb6a41061b0 2024-03-12T03:08:52.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.848+0000 7fb6a37fe700 1 -- 172.21.15.94:0/4230820852 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fb698004930 con 0x7fb6a41061b0 2024-03-12T03:08:52.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.848+0000 7fb6a37fe700 1 -- 172.21.15.94:0/4230820852 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fb698005a70 con 0x7fb6a41061b0 2024-03-12T03:08:52.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.849+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/4230820852 >> v1:172.21.15.94:6789/0 conn(0x7fb6a41061b0 legacy=0x7fb6a41065d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:52.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.849+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/4230820852 shutdown_connections 2024-03-12T03:08:52.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.849+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/4230820852 >> 172.21.15.94:0/4230820852 conn(0x7fb6a40ff8e0 msgr2=0x7fb6a4101d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:52.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.849+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/4230820852 shutdown_connections 2024-03-12T03:08:52.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.849+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/4230820852 wait complete. 2024-03-12T03:08:52.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.850+0000 7fb6aaf2f700 1 Processor -- start 2024-03-12T03:08:52.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.850+0000 7fb6aaf2f700 1 -- start start 2024-03-12T03:08:52.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.850+0000 7fb6aaf2f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6a4104780 con 0x7fb6a41061b0 2024-03-12T03:08:52.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.850+0000 7fb6a8ccb700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb6a41061b0 0x7fb6a419c590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56144/0 (socket says 172.21.15.94:56144) 2024-03-12T03:08:52.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.850+0000 7fb6a8ccb700 1 -- 172.21.15.94:0/2056561609 learned_addr learned my addr 172.21.15.94:0/2056561609 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:52.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.850+0000 7fb6a1ffb700 1 -- 172.21.15.94:0/2056561609 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668644073 0 0) 0x7fb6a4104780 con 0x7fb6a41061b0 2024-03-12T03:08:52.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.850+0000 7fb6a1ffb700 1 -- 172.21.15.94:0/2056561609 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb694005530 con 0x7fb6a41061b0 2024-03-12T03:08:52.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.851+0000 7fb6a1ffb700 1 -- 172.21.15.94:0/2056561609 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2053463582 0 0) 0x7fb694005530 con 0x7fb6a41061b0 2024-03-12T03:08:52.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.851+0000 7fb6a1ffb700 1 -- 172.21.15.94:0/2056561609 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6a4104780 con 0x7fb6a41061b0 2024-03-12T03:08:52.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.851+0000 7fb6a1ffb700 1 -- 172.21.15.94:0/2056561609 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fb698001080 con 0x7fb6a41061b0 2024-03-12T03:08:52.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.851+0000 7fb6a1ffb700 1 -- 172.21.15.94:0/2056561609 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 225291788 0 0) 0x7fb6a4104780 con 0x7fb6a41061b0 2024-03-12T03:08:52.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.851+0000 7fb6a1ffb700 1 -- 172.21.15.94:0/2056561609 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a4104950 con 0x7fb6a41061b0 2024-03-12T03:08:52.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.851+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/2056561609 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a4104c40 con 0x7fb6a41061b0 2024-03-12T03:08:52.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.851+0000 7fb6a1ffb700 1 -- 172.21.15.94:0/2056561609 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fb6980059c0 con 0x7fb6a41061b0 2024-03-12T03:08:52.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.851+0000 7fb6a1ffb700 1 -- 172.21.15.94:0/2056561609 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fb698006370 con 0x7fb6a41061b0 2024-03-12T03:08:52.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.851+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/2056561609 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a419df60 con 0x7fb6a41061b0 2024-03-12T03:08:52.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.852+0000 7fb6a1ffb700 1 -- 172.21.15.94:0/2056561609 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 8) v1 ==== 46071+0+0 (unknown 2158625433 0 0) 0x7fb698011a60 con 0x7fb6a41061b0 2024-03-12T03:08:52.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.852+0000 7fb6a1ffb700 1 -- 172.21.15.94:0/2056561609 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 4125034817 0 0) 0x7fb698001b60 con 0x7fb6a41061b0 2024-03-12T03:08:52.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.852+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/2056561609 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a4063410 con 0x7fb6a41061b0 2024-03-12T03:08:52.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:51.857+0000 7fb6a1ffb700 1 -- 172.21.15.94:0/2056561609 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fb6980175f0 con 0x7fb6a41061b0 2024-03-12T03:08:52.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:52.070+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/2056561609 --> v1:172.21.15.94:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fb6a4191be0 con 0x7fb6a41061b0 2024-03-12T03:08:52.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:52.073+0000 7fb6a1ffb700 1 -- 172.21.15.94:0/2056561609 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (unknown 2684808024 0 0) 0x7fb6a4191be0 con 0x7fb6a41061b0 2024-03-12T03:08:52.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:52.075+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/2056561609 >> v1:172.21.15.94:6800/3548280253 conn(0x7fb69403a410 legacy=0x7fb69403c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:52.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:52.075+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/2056561609 >> v1:172.21.15.94:6789/0 conn(0x7fb6a41061b0 legacy=0x7fb6a419c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:52.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:52.075+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/2056561609 shutdown_connections 2024-03-12T03:08:52.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:52.075+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/2056561609 >> 172.21.15.94:0/2056561609 conn(0x7fb6a40ff8e0 msgr2=0x7fb6a4101d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:52.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:52.075+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/2056561609 shutdown_connections 2024-03-12T03:08:52.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:52.075+0000 7fb6aaf2f700 1 -- 172.21.15.94:0/2056561609 wait complete. 2024-03-12T03:08:53.355 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2056561609' entity='client.admin' 2024-03-12T03:08:53.355 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:53 smithi094 ceph-mon[29291]: from='mgr.14120 v1:172.21.15.94:0/615254318' entity='mgr.y' 2024-03-12T03:08:54.042 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.202+0000 7f3f90f5c700 1 Processor -- start 2024-03-12T03:08:54.042 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.202+0000 7f3f90f5c700 1 -- start start 2024-03-12T03:08:54.042 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.202+0000 7f3f90f5c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3f8c10a1b0 con 0x7f3f8c103ff0 2024-03-12T03:08:54.042 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.203+0000 7f3f8a59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3f8c103ff0 0x7f3f8c104410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56160/0 (socket says 172.21.15.94:56160) 2024-03-12T03:08:54.042 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.203+0000 7f3f8a59c700 1 -- 172.21.15.94:0/198638081 learned_addr learned my addr 172.21.15.94:0/198638081 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:54.042 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.203+0000 7f3f8959a700 1 -- 172.21.15.94:0/198638081 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68759064 0 0) 0x7f3f8c10a1b0 con 0x7f3f8c103ff0 2024-03-12T03:08:54.042 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.203+0000 7f3f8959a700 1 -- 172.21.15.94:0/198638081 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f740035b0 con 0x7f3f8c103ff0 2024-03-12T03:08:54.042 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.203+0000 7f3f8959a700 1 -- 172.21.15.94:0/198638081 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 221363761 0 0) 0x7f3f740035b0 con 0x7f3f8c103ff0 2024-03-12T03:08:54.042 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.203+0000 7f3f8959a700 1 -- 172.21.15.94:0/198638081 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8c10b390 con 0x7f3f8c103ff0 2024-03-12T03:08:54.043 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.203+0000 7f3f8959a700 1 -- 172.21.15.94:0/198638081 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f3f7c001580 con 0x7f3f8c103ff0 2024-03-12T03:08:54.043 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.203+0000 7f3f8959a700 1 -- 172.21.15.94:0/198638081 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f3f7c004930 con 0x7f3f8c103ff0 2024-03-12T03:08:54.043 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.203+0000 7f3f8959a700 1 -- 172.21.15.94:0/198638081 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f3f7c005a70 con 0x7f3f8c103ff0 2024-03-12T03:08:54.043 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.204+0000 7f3f90f5c700 1 -- 172.21.15.94:0/198638081 >> v1:172.21.15.94:6789/0 conn(0x7f3f8c103ff0 legacy=0x7f3f8c104410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:54.043 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.204+0000 7f3f90f5c700 1 -- 172.21.15.94:0/198638081 shutdown_connections 2024-03-12T03:08:54.043 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.204+0000 7f3f90f5c700 1 -- 172.21.15.94:0/198638081 >> 172.21.15.94:0/198638081 conn(0x7f3f8c0ff8e0 msgr2=0x7f3f8c101d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:54.043 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.204+0000 7f3f90f5c700 1 -- 172.21.15.94:0/198638081 shutdown_connections 2024-03-12T03:08:54.043 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.204+0000 7f3f90f5c700 1 -- 172.21.15.94:0/198638081 wait complete. 2024-03-12T03:08:54.043 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.205+0000 7f3f90f5c700 1 Processor -- start 2024-03-12T03:08:54.043 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.205+0000 7f3f90f5c700 1 -- start start 2024-03-12T03:08:54.043 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.205+0000 7f3f90f5c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3f8c199f80 con 0x7f3f8c103ff0 2024-03-12T03:08:54.043 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.205+0000 7f3f8a59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3f8c103ff0 0x7f3f8c199870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56164/0 (socket says 172.21.15.94:56164) 2024-03-12T03:08:54.043 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.205+0000 7f3f8a59c700 1 -- 172.21.15.94:0/3517910835 learned_addr learned my addr 172.21.15.94:0/3517910835 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:54.044 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.205+0000 7f3f7b7fe700 1 -- 172.21.15.94:0/3517910835 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492777927 0 0) 0x7f3f8c199f80 con 0x7f3f8c103ff0 2024-03-12T03:08:54.044 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.206+0000 7f3f7b7fe700 1 -- 172.21.15.94:0/3517910835 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f74005530 con 0x7f3f8c103ff0 2024-03-12T03:08:54.044 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.206+0000 7f3f7b7fe700 1 -- 172.21.15.94:0/3517910835 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2983682656 0 0) 0x7f3f74005530 con 0x7f3f8c103ff0 2024-03-12T03:08:54.044 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.206+0000 7f3f7b7fe700 1 -- 172.21.15.94:0/3517910835 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f8c199f80 con 0x7f3f8c103ff0 2024-03-12T03:08:54.044 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.206+0000 7f3f7b7fe700 1 -- 172.21.15.94:0/3517910835 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f3f7c001080 con 0x7f3f8c103ff0 2024-03-12T03:08:54.044 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.206+0000 7f3f7b7fe700 1 -- 172.21.15.94:0/3517910835 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3382333590 0 0) 0x7f3f8c199f80 con 0x7f3f8c103ff0 2024-03-12T03:08:54.044 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.206+0000 7f3f7b7fe700 1 -- 172.21.15.94:0/3517910835 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8c19a150 con 0x7f3f8c103ff0 2024-03-12T03:08:54.044 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.206+0000 7f3f90f5c700 1 -- 172.21.15.94:0/3517910835 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8c19a440 con 0x7f3f8c103ff0 2024-03-12T03:08:54.044 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.206+0000 7f3f90f5c700 1 -- 172.21.15.94:0/3517910835 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8c19df90 con 0x7f3f8c103ff0 2024-03-12T03:08:54.044 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.206+0000 7f3f7b7fe700 1 -- 172.21.15.94:0/3517910835 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f3f7c005a60 con 0x7f3f8c103ff0 2024-03-12T03:08:54.044 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.206+0000 7f3f7b7fe700 1 -- 172.21.15.94:0/3517910835 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f3f7c0062c0 con 0x7f3f8c103ff0 2024-03-12T03:08:54.044 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.207+0000 7f3f7b7fe700 1 -- 172.21.15.94:0/3517910835 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 8) v1 ==== 46071+0+0 (unknown 2158625433 0 0) 0x7f3f7c011860 con 0x7f3f8c103ff0 2024-03-12T03:08:54.044 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.207+0000 7f3f90f5c700 1 -- 172.21.15.94:0/3517910835 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6c005150 con 0x7f3f8c103ff0 2024-03-12T03:08:54.045 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.207+0000 7f3f7b7fe700 1 -- 172.21.15.94:0/3517910835 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 4125034817 0 0) 0x7f3f7c0487e0 con 0x7f3f8c103ff0 2024-03-12T03:08:54.045 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.213+0000 7f3f7b7fe700 1 -- 172.21.15.94:0/3517910835 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f3f7c017770 con 0x7f3f8c103ff0 2024-03-12T03:08:54.045 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.426+0000 7f3f90f5c700 1 -- 172.21.15.94:0/3517910835 --> v1:172.21.15.94:6789/0 -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f3f6c005440 con 0x7f3f8c103ff0 2024-03-12T03:08:54.045 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.428+0000 7f3f7b7fe700 1 -- 172.21.15.94:0/3517910835 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (unknown 1496729260 0 0) 0x7f3f7c017070 con 0x7f3f8c103ff0 2024-03-12T03:08:54.045 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.430+0000 7f3f90f5c700 1 -- 172.21.15.94:0/3517910835 >> v1:172.21.15.94:6800/3548280253 conn(0x7f3f7403a460 legacy=0x7f3f7403c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:54.045 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.430+0000 7f3f90f5c700 1 -- 172.21.15.94:0/3517910835 >> v1:172.21.15.94:6789/0 conn(0x7f3f8c103ff0 legacy=0x7f3f8c199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:54.045 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.431+0000 7f3f90f5c700 1 -- 172.21.15.94:0/3517910835 shutdown_connections 2024-03-12T03:08:54.045 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.431+0000 7f3f90f5c700 1 -- 172.21.15.94:0/3517910835 >> 172.21.15.94:0/3517910835 conn(0x7f3f8c0ff8e0 msgr2=0x7f3f8c101d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:54.045 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.431+0000 7f3f90f5c700 1 -- 172.21.15.94:0/3517910835 shutdown_connections 2024-03-12T03:08:54.045 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:53.431+0000 7f3f90f5c700 1 -- 172.21.15.94:0/3517910835 wait complete. 2024-03-12T03:08:54.045 INFO:teuthology.orchestra.run.smithi094.stdout:Enabling the dashboard module... 2024-03-12T03:08:54.630 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3517910835' entity='client.admin' 2024-03-12T03:08:55.696 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2155573353' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-12T03:08:55.696 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:55 smithi094 ceph-mon[29291]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:08:55.697 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:55 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ignoring --setuser ceph since I am not root 2024-03-12T03:08:55.697 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:55 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ignoring --setgroup ceph since I am not root 2024-03-12T03:08:56.022 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.451+0000 7f4334d86700 1 Processor -- start 2024-03-12T03:08:56.022 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.451+0000 7f4334d86700 1 -- start start 2024-03-12T03:08:56.022 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.452+0000 7f4334d86700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4330078e20 con 0x7f4330106790 2024-03-12T03:08:56.022 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.452+0000 7f432e59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4330106790 0x7f4330108b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42898/0 (socket says 172.21.15.94:42898) 2024-03-12T03:08:56.022 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.452+0000 7f432e59c700 1 -- 172.21.15.94:0/1789905064 learned_addr learned my addr 172.21.15.94:0/1789905064 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.452+0000 7f432d59a700 1 -- 172.21.15.94:0/1789905064 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969600908 0 0) 0x7f4330078e20 con 0x7f4330106790 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.452+0000 7f432d59a700 1 -- 172.21.15.94:0/1789905064 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43180035b0 con 0x7f4330106790 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.452+0000 7f432d59a700 1 -- 172.21.15.94:0/1789905064 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1306707504 0 0) 0x7f43180035b0 con 0x7f4330106790 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.452+0000 7f432d59a700 1 -- 172.21.15.94:0/1789905064 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433010b2b0 con 0x7f4330106790 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.452+0000 7f432d59a700 1 -- 172.21.15.94:0/1789905064 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f4320001580 con 0x7f4330106790 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.453+0000 7f432d59a700 1 -- 172.21.15.94:0/1789905064 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f4320004930 con 0x7f4330106790 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.453+0000 7f432d59a700 1 -- 172.21.15.94:0/1789905064 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f4320005a70 con 0x7f4330106790 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.453+0000 7f4334d86700 1 -- 172.21.15.94:0/1789905064 >> v1:172.21.15.94:6789/0 conn(0x7f4330106790 legacy=0x7f4330108b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.453+0000 7f4334d86700 1 -- 172.21.15.94:0/1789905064 shutdown_connections 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.453+0000 7f4334d86700 1 -- 172.21.15.94:0/1789905064 >> 172.21.15.94:0/1789905064 conn(0x7f4330100160 msgr2=0x7f43301025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.453+0000 7f4334d86700 1 -- 172.21.15.94:0/1789905064 shutdown_connections 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.453+0000 7f4334d86700 1 -- 172.21.15.94:0/1789905064 wait complete. 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.454+0000 7f4334d86700 1 Processor -- start 2024-03-12T03:08:56.023 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.454+0000 7f4334d86700 1 -- start start 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.454+0000 7f4334d86700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4330199f60 con 0x7f4330106790 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.454+0000 7f432e59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4330106790 0x7f4330199850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42902/0 (socket says 172.21.15.94:42902) 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.454+0000 7f432e59c700 1 -- 172.21.15.94:0/2155573353 learned_addr learned my addr 172.21.15.94:0/2155573353 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.454+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792800877 0 0) 0x7f4330199f60 con 0x7f4330106790 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.454+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4318005530 con 0x7f4330106790 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.455+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 561007598 0 0) 0x7f4318005530 con 0x7f4330106790 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.455+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4330199f60 con 0x7f4330106790 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.455+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f4320001080 con 0x7f4330106790 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.455+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1802097590 0 0) 0x7f4330199f60 con 0x7f4330106790 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.455+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433019a130 con 0x7f4330106790 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.455+0000 7f4334d86700 1 -- 172.21.15.94:0/2155573353 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433019a420 con 0x7f4330106790 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.455+0000 7f4334d86700 1 -- 172.21.15.94:0/2155573353 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f433019df70 con 0x7f4330106790 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.455+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f4320004930 con 0x7f4330106790 2024-03-12T03:08:56.024 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.455+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f4320006130 con 0x7f4330106790 2024-03-12T03:08:56.025 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.456+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 8) v1 ==== 46071+0+0 (unknown 2158625433 0 0) 0x7f43200117f0 con 0x7f4330106790 2024-03-12T03:08:56.025 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.456+0000 7f4334d86700 1 -- 172.21.15.94:0/2155573353 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4310005120 con 0x7f4330106790 2024-03-12T03:08:56.025 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.456+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 4125034817 0 0) 0x7f43200479e0 con 0x7f4330106790 2024-03-12T03:08:56.025 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.461+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f4320017740 con 0x7f4330106790 2024-03-12T03:08:56.025 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:54.717+0000 7f4334d86700 1 -- 172.21.15.94:0/2155573353 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f4310005410 con 0x7f4330106790 2024-03-12T03:08:56.025 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:55.430+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (unknown 1498667528 0 0) 0x7f4320017040 con 0x7f4330106790 2024-03-12T03:08:56.025 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:55.432+0000 7f431f7fe700 1 -- 172.21.15.94:0/2155573353 <== mon.0 v1:172.21.15.94:6789/0 11 ==== mgrmap(e 9) v1 ==== 46084+0+0 (unknown 1502715055 0 0) 0x7f4320046320 con 0x7f4330106790 2024-03-12T03:08:56.025 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:55.434+0000 7f4334d86700 1 -- 172.21.15.94:0/2155573353 >> v1:172.21.15.94:6800/3548280253 conn(0x7f431803a320 legacy=0x7f431803c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:56.025 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:55.434+0000 7f4334d86700 1 -- 172.21.15.94:0/2155573353 >> v1:172.21.15.94:6789/0 conn(0x7f4330106790 legacy=0x7f4330199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:56.025 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:55.434+0000 7f4334d86700 1 -- 172.21.15.94:0/2155573353 shutdown_connections 2024-03-12T03:08:56.025 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:55.434+0000 7f4334d86700 1 -- 172.21.15.94:0/2155573353 >> 172.21.15.94:0/2155573353 conn(0x7f4330100160 msgr2=0x7f4330190510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:56.025 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:55.434+0000 7f4334d86700 1 -- 172.21.15.94:0/2155573353 shutdown_connections 2024-03-12T03:08:56.025 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:55.434+0000 7f4334d86700 1 -- 172.21.15.94:0/2155573353 wait complete. 2024-03-12T03:08:56.260 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:56 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:56.000+0000 7f86f608d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:08:56.260 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:56 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:56.098+0000 7f86f608d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:08:56.657 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2155573353' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-12T03:08:56.658 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:56 smithi094 ceph-mon[29291]: mgrmap e9: y(active, since 20s) 2024-03-12T03:08:56.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:56 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:56.760+0000 7f86f608d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:08:56.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:56 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:56.853+0000 7f86f608d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:08:57.372 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:56 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:56.997+0000 7f86f608d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:08:57.373 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:57 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:57.094+0000 7f86f608d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:08:57.373 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:57 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:57.179+0000 7f86f608d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:08:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:08:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-12T03:08:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T03:08:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-12T03:08:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T03:08:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:08:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.449+0000 7f0760f71700 1 Processor -- start 2024-03-12T03:08:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.449+0000 7f0760f71700 1 -- start start 2024-03-12T03:08:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.449+0000 7f0760f71700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f075c07afe0 con 0x7f075c0792b0 2024-03-12T03:08:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.449+0000 7f075a59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f075c0792b0 0x7f075c0796d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42930/0 (socket says 172.21.15.94:42930) 2024-03-12T03:08:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.449+0000 7f075a59c700 1 -- 172.21.15.94:0/1746536270 learned_addr learned my addr 172.21.15.94:0/1746536270 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.449+0000 7f075959a700 1 -- 172.21.15.94:0/1746536270 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974339854 0 0) 0x7f075c07afe0 con 0x7f075c0792b0 2024-03-12T03:08:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.449+0000 7f075959a700 1 -- 172.21.15.94:0/1746536270 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07440035b0 con 0x7f075c0792b0 2024-03-12T03:08:57.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.450+0000 7f075959a700 1 -- 172.21.15.94:0/1746536270 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3918742086 0 0) 0x7f07440035b0 con 0x7f075c0792b0 2024-03-12T03:08:57.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.450+0000 7f075959a700 1 -- 172.21.15.94:0/1746536270 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c079de0 con 0x7f075c0792b0 2024-03-12T03:08:57.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.450+0000 7f075959a700 1 -- 172.21.15.94:0/1746536270 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f074c001580 con 0x7f075c0792b0 2024-03-12T03:08:57.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.450+0000 7f075959a700 1 -- 172.21.15.94:0/1746536270 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f074c001c70 con 0x7f075c0792b0 2024-03-12T03:08:57.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.450+0000 7f075959a700 1 -- 172.21.15.94:0/1746536270 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f074c005960 con 0x7f075c0792b0 2024-03-12T03:08:57.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.451+0000 7f0760f71700 1 -- 172.21.15.94:0/1746536270 >> v1:172.21.15.94:6789/0 conn(0x7f075c0792b0 legacy=0x7f075c0796d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:57.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.451+0000 7f0760f71700 1 -- 172.21.15.94:0/1746536270 shutdown_connections 2024-03-12T03:08:57.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.451+0000 7f0760f71700 1 -- 172.21.15.94:0/1746536270 >> 172.21.15.94:0/1746536270 conn(0x7f075c101290 msgr2=0x7f075c103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:57.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.451+0000 7f0760f71700 1 -- 172.21.15.94:0/1746536270 shutdown_connections 2024-03-12T03:08:57.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.451+0000 7f0760f71700 1 -- 172.21.15.94:0/1746536270 wait complete. 2024-03-12T03:08:57.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.452+0000 7f0760f71700 1 Processor -- start 2024-03-12T03:08:57.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.452+0000 7f0760f71700 1 -- start start 2024-03-12T03:08:57.382 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.452+0000 7f0760f71700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f075c19c310 con 0x7f075c0792b0 2024-03-12T03:08:57.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.452+0000 7f075a59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f075c0792b0 0x7f075c19bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42940/0 (socket says 172.21.15.94:42940) 2024-03-12T03:08:57.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.452+0000 7f075a59c700 1 -- 172.21.15.94:0/3140927467 learned_addr learned my addr 172.21.15.94:0/3140927467 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:08:57.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.452+0000 7f074b7fe700 1 -- 172.21.15.94:0/3140927467 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346417721 0 0) 0x7f075c19c310 con 0x7f075c0792b0 2024-03-12T03:08:57.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.452+0000 7f074b7fe700 1 -- 172.21.15.94:0/3140927467 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0744005580 con 0x7f075c0792b0 2024-03-12T03:08:57.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.452+0000 7f074b7fe700 1 -- 172.21.15.94:0/3140927467 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 482446073 0 0) 0x7f0744005580 con 0x7f075c0792b0 2024-03-12T03:08:57.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.452+0000 7f074b7fe700 1 -- 172.21.15.94:0/3140927467 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f075c19c310 con 0x7f075c0792b0 2024-03-12T03:08:57.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.453+0000 7f074b7fe700 1 -- 172.21.15.94:0/3140927467 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f074c001080 con 0x7f075c0792b0 2024-03-12T03:08:57.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.453+0000 7f074b7fe700 1 -- 172.21.15.94:0/3140927467 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1796403157 0 0) 0x7f075c19c310 con 0x7f075c0792b0 2024-03-12T03:08:57.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.453+0000 7f074b7fe700 1 -- 172.21.15.94:0/3140927467 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c19c4e0 con 0x7f075c0792b0 2024-03-12T03:08:57.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.453+0000 7f0760f71700 1 -- 172.21.15.94:0/3140927467 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075c19c7d0 con 0x7f075c0792b0 2024-03-12T03:08:57.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.453+0000 7f0760f71700 1 -- 172.21.15.94:0/3140927467 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f075c1a0320 con 0x7f075c0792b0 2024-03-12T03:08:57.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.453+0000 7f074b7fe700 1 -- 172.21.15.94:0/3140927467 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f074c001c30 con 0x7f075c0792b0 2024-03-12T03:08:57.383 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.453+0000 7f074b7fe700 1 -- 172.21.15.94:0/3140927467 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f074c006040 con 0x7f075c0792b0 2024-03-12T03:08:57.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.454+0000 7f074b7fe700 1 -- 172.21.15.94:0/3140927467 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 9) v1 ==== 46084+0+0 (unknown 1502715055 0 0) 0x7f074c0116d0 con 0x7f075c0792b0 2024-03-12T03:08:57.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.454+0000 7f0760f71700 1 -- 172.21.15.94:0/3140927467 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073c005150 con 0x7f075c0792b0 2024-03-12T03:08:57.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.454+0000 7f074b7fe700 1 -- 172.21.15.94:0/3140927467 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 4125034817 0 0) 0x7f074c047900 con 0x7f075c0792b0 2024-03-12T03:08:57.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.454+0000 7f0759d9b700 1 -- 172.21.15.94:0/3140927467 >> v1:172.21.15.94:6800/3548280253 conn(0x7f074403a3e0 legacy=0x7f074403c8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/3548280253 2024-03-12T03:08:57.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.458+0000 7f074b7fe700 1 -- 172.21.15.94:0/3140927467 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f074c017660 con 0x7f075c0792b0 2024-03-12T03:08:57.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.655+0000 7f0759d9b700 1 -- 172.21.15.94:0/3140927467 >> v1:172.21.15.94:6800/3548280253 conn(0x7f074403a3e0 legacy=0x7f074403c8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/3548280253 2024-03-12T03:08:57.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.715+0000 7f0760f71700 1 -- 172.21.15.94:0/3140927467 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f073c005c50 con 0x7f075c0792b0 2024-03-12T03:08:57.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.716+0000 7f074b7fe700 1 -- 172.21.15.94:0/3140927467 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (unknown 2005831594 0 1748205903) 0x7f074c016f60 con 0x7f075c0792b0 2024-03-12T03:08:57.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.718+0000 7f0760f71700 1 -- 172.21.15.94:0/3140927467 >> v1:172.21.15.94:6800/3548280253 conn(0x7f074403a3e0 legacy=0x7f074403c8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T03:08:57.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.718+0000 7f0760f71700 1 -- 172.21.15.94:0/3140927467 >> v1:172.21.15.94:6789/0 conn(0x7f075c0792b0 legacy=0x7f075c19bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:08:57.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.718+0000 7f0760f71700 1 -- 172.21.15.94:0/3140927467 shutdown_connections 2024-03-12T03:08:57.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.718+0000 7f0760f71700 1 -- 172.21.15.94:0/3140927467 >> 172.21.15.94:0/3140927467 conn(0x7f075c101290 msgr2=0x7f075c10a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:08:57.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.718+0000 7f0760f71700 1 -- 172.21.15.94:0/3140927467 shutdown_connections 2024-03-12T03:08:57.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:56.718+0000 7f0760f71700 1 -- 172.21.15.94:0/3140927467 wait complete. 2024-03-12T03:08:57.385 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for the mgr to restart... 2024-03-12T03:08:57.385 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mgr epoch 9... 2024-03-12T03:08:57.628 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:08:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3140927467' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T03:08:57.628 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:57 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:57.450+0000 7f86f608d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:08:57.628 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:57 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:57.562+0000 7f86f608d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:08:57.932 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:57 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:57.652+0000 7f86f608d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:08:57.932 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:57 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:57.747+0000 7f86f608d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:08:58.209 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:57 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:57.929+0000 7f86f608d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:08:59.209 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:58.894+0000 7f86f608d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:08:59.209 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:59 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:59.027+0000 7f86f608d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:08:59.652 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:59 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:59.361+0000 7f86f608d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:08:59.652 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:59 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:59.452+0000 7f86f608d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:08:59.652 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:59 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:59.649+0000 7f86f608d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:08:59.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:08:59 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:08:59.837+0000 7f86f608d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:09:00.383 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:09:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:09:00.123+0000 7f86f608d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:09:00.384 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:09:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:09:00.381+0000 7f86f608d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:09:00.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:09:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:09:00.619+0000 7f86f608d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:09:01.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:09:01 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:09:01.422+0000 7f86f608d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:09:02.304 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:09:01 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:09:01.988+0000 7f86f608d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:09:02.304 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:09:02.086+0000 7f86f608d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:09:02.304 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:09:02.301+0000 7f86f608d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:09:02.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-mon[29291]: Active manager daemon y restarted 2024-03-12T03:09:02.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-mon[29291]: Activating manager daemon y 2024-03-12T03:09:02.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-mon[29291]: osdmap e3: 0 total, 0 up, 0 in 2024-03-12T03:09:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-mon[29291]: mgrmap e10: y(active, starting, since 0.00314086s) 2024-03-12T03:09:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:09:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:09:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:09:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:09:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:09:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-mon[29291]: Manager daemon y is now available 2024-03-12T03:09:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:09:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:09:03.550 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:03 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:09:03.550 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:03 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:09:03.553 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:03 smithi094 ceph-mon[29291]: mgrmap e11: y(active, since 1.00546s) 2024-03-12T03:09:04.444 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:04 smithi094 ceph-mon[29291]: [12/Mar/2024:03:09:02] ENGINE Bus STARTING 2024-03-12T03:09:04.444 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:04 smithi094 ceph-mon[29291]: [12/Mar/2024:03:09:02] ENGINE Serving on http://172.21.15.94:8765 2024-03-12T03:09:04.444 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:04 smithi094 ceph-mon[29291]: [12/Mar/2024:03:09:02] ENGINE Serving on https://172.21.15.94:7150 2024-03-12T03:09:04.444 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:04 smithi094 ceph-mon[29291]: [12/Mar/2024:03:09:02] ENGINE Bus STARTED 2024-03-12T03:09:04.445 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:04 smithi094 ceph-mon[29291]: from='client.14156 v1:172.21.15.94:0/3028456359' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T03:09:04.445 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:04 smithi094 ceph-mon[29291]: from='client.14156 v1:172.21.15.94:0/3028456359' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T03:09:04.445 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:04 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:04.452 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:09:04.452 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-12T03:09:04.452 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T03:09:04.452 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:09:04.453 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.868+0000 7fbafa565700 1 Processor -- start 2024-03-12T03:09:04.453 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.868+0000 7fbafa565700 1 -- start start 2024-03-12T03:09:04.453 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.868+0000 7fbafa565700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbaf4078e00 con 0x7fbaf41045a0 2024-03-12T03:09:04.453 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.869+0000 7fbaf3fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbaf41045a0 0x7fbaf41049c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42950/0 (socket says 172.21.15.94:42950) 2024-03-12T03:09:04.453 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.869+0000 7fbaf3fff700 1 -- 172.21.15.94:0/2592737738 learned_addr learned my addr 172.21.15.94:0/2592737738 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:04.453 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.869+0000 7fbaf2ffd700 1 -- 172.21.15.94:0/2592737738 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659257627 0 0) 0x7fbaf4078e00 con 0x7fbaf41045a0 2024-03-12T03:09:04.453 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.869+0000 7fbaf2ffd700 1 -- 172.21.15.94:0/2592737738 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae00035b0 con 0x7fbaf41045a0 2024-03-12T03:09:04.453 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.869+0000 7fbaf2ffd700 1 -- 172.21.15.94:0/2592737738 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1906615267 0 0) 0x7fbae00035b0 con 0x7fbaf41045a0 2024-03-12T03:09:04.453 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.869+0000 7fbaf2ffd700 1 -- 172.21.15.94:0/2592737738 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf41060e0 con 0x7fbaf41045a0 2024-03-12T03:09:04.453 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.869+0000 7fbaf2ffd700 1 -- 172.21.15.94:0/2592737738 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fbadc001580 con 0x7fbaf41045a0 2024-03-12T03:09:04.453 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.869+0000 7fbaf2ffd700 1 -- 172.21.15.94:0/2592737738 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fbadc004930 con 0x7fbaf41045a0 2024-03-12T03:09:04.453 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.869+0000 7fbaf2ffd700 1 -- 172.21.15.94:0/2592737738 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fbadc005a70 con 0x7fbaf41045a0 2024-03-12T03:09:04.453 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.870+0000 7fbafa565700 1 -- 172.21.15.94:0/2592737738 >> v1:172.21.15.94:6789/0 conn(0x7fbaf41045a0 legacy=0x7fbaf41049c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.870+0000 7fbafa565700 1 -- 172.21.15.94:0/2592737738 shutdown_connections 2024-03-12T03:09:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.870+0000 7fbafa565700 1 -- 172.21.15.94:0/2592737738 >> 172.21.15.94:0/2592737738 conn(0x7fbaf4100160 msgr2=0x7fbaf41025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.870+0000 7fbafa565700 1 -- 172.21.15.94:0/2592737738 shutdown_connections 2024-03-12T03:09:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.870+0000 7fbafa565700 1 -- 172.21.15.94:0/2592737738 wait complete. 2024-03-12T03:09:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.871+0000 7fbafa565700 1 Processor -- start 2024-03-12T03:09:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.871+0000 7fbafa565700 1 -- start start 2024-03-12T03:09:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.871+0000 7fbafa565700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbaf419c140 con 0x7fbaf41045a0 2024-03-12T03:09:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.871+0000 7fbaf3fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbaf41045a0 0x7fbaf419ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42958/0 (socket says 172.21.15.94:42958) 2024-03-12T03:09:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.871+0000 7fbaf3fff700 1 -- 172.21.15.94:0/3028456359 learned_addr learned my addr 172.21.15.94:0/3028456359 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.871+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486408645 0 0) 0x7fbaf419c140 con 0x7fbaf41045a0 2024-03-12T03:09:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.871+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae0005530 con 0x7fbaf41045a0 2024-03-12T03:09:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.872+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3512270621 0 0) 0x7fbae0005530 con 0x7fbaf41045a0 2024-03-12T03:09:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.872+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbaf419c140 con 0x7fbaf41045a0 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.872+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fbadc001080 con 0x7fbaf41045a0 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.872+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2537193248 0 0) 0x7fbaf419c140 con 0x7fbaf41045a0 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.872+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf419c310 con 0x7fbaf41045a0 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.872+0000 7fbafa565700 1 -- 172.21.15.94:0/3028456359 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf419c600 con 0x7fbaf41045a0 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.872+0000 7fbafa565700 1 -- 172.21.15.94:0/3028456359 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf41a0150 con 0x7fbaf41045a0 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.872+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fbadc004930 con 0x7fbaf41045a0 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.872+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fbadc006050 con 0x7fbaf41045a0 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.873+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 9) v1 ==== 46084+0+0 (unknown 1502715055 0 0) 0x7fbadc011620 con 0x7fbaf41045a0 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.873+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 --> v1:172.21.15.94:6800/3548280253 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbadc001080 con 0x7fbae003a460 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.873+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 4125034817 0 0) 0x7fbadc048630 con 0x7fbaf41045a0 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:57.873+0000 7fbaf37fe700 1 -- 172.21.15.94:0/3028456359 >> v1:172.21.15.94:6800/3548280253 conn(0x7fbae003a460 legacy=0x7fbae003c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/3548280253 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:58.074+0000 7fbaf37fe700 1 -- 172.21.15.94:0/3028456359 >> v1:172.21.15.94:6800/3548280253 conn(0x7fbae003a460 legacy=0x7fbae003c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/3548280253 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:58.475+0000 7fbaf37fe700 1 -- 172.21.15.94:0/3028456359 >> v1:172.21.15.94:6800/3548280253 conn(0x7fbae003a460 legacy=0x7fbae003c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/3548280253 2024-03-12T03:09:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:08:59.276+0000 7fbaf37fe700 1 -- 172.21.15.94:0/3028456359 >> v1:172.21.15.94:6800/3548280253 conn(0x7fbae003a460 legacy=0x7fbae003c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/3548280253 2024-03-12T03:09:04.456 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:00.877+0000 7fbaf37fe700 1 -- 172.21.15.94:0/3028456359 >> v1:172.21.15.94:6800/3548280253 conn(0x7fbae003a460 legacy=0x7fbae003c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/3548280253 2024-03-12T03:09:04.456 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:02.316+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mgrmap(e 10) v1 ==== 45886+0+0 (unknown 3941242571 0 0) 0x7fbadc012be0 con 0x7fbaf41045a0 2024-03-12T03:09:04.456 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:02.316+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 >> v1:172.21.15.94:6800/3548280253 conn(0x7fbae003a460 legacy=0x7fbae003c920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T03:09:04.456 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:03.317+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mgrmap(e 11) v1 ==== 45978+0+0 (unknown 1807387427 0 0) 0x7fbadc049270 con 0x7fbaf41045a0 2024-03-12T03:09:04.456 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:03.317+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 --> v1:172.21.15.94:6800/1299282424 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbadc001080 con 0x7fbae003ee10 2024-03-12T03:09:04.456 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:03.318+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (unknown 0 0 4109177037) 0x7fbadc001080 con 0x7fbae003ee10 2024-03-12T03:09:04.456 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:03.325+0000 7fbafa565700 1 -- 172.21.15.94:0/3028456359 --> v1:172.21.15.94:6800/1299282424 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fbad4002630 con 0x7fbae003ee10 2024-03-12T03:09:04.456 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:03.325+0000 7fbaf17fa700 1 -- 172.21.15.94:0/3028456359 <== mgr.14152 v1:172.21.15.94:6800/1299282424 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (unknown 0 0 3086460295) 0x7fbad4002630 con 0x7fbae003ee10 2024-03-12T03:09:04.456 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:03.325+0000 7fbafa565700 1 -- 172.21.15.94:0/3028456359 >> v1:172.21.15.94:6800/1299282424 conn(0x7fbae003ee10 legacy=0x7fbae0041200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:04.456 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:03.325+0000 7fbafa565700 1 -- 172.21.15.94:0/3028456359 >> v1:172.21.15.94:6789/0 conn(0x7fbaf41045a0 legacy=0x7fbaf419ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:04.456 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:03.325+0000 7fbafa565700 1 -- 172.21.15.94:0/3028456359 shutdown_connections 2024-03-12T03:09:04.456 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:03.325+0000 7fbafa565700 1 -- 172.21.15.94:0/3028456359 >> 172.21.15.94:0/3028456359 conn(0x7fbaf4100160 msgr2=0x7fbaf41025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:04.457 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:03.325+0000 7fbafa565700 1 -- 172.21.15.94:0/3028456359 shutdown_connections 2024-03-12T03:09:04.457 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:03.325+0000 7fbafa565700 1 -- 172.21.15.94:0/3028456359 wait complete. 2024-03-12T03:09:04.457 INFO:teuthology.orchestra.run.smithi094.stdout:mgr epoch 9 is available 2024-03-12T03:09:04.457 INFO:teuthology.orchestra.run.smithi094.stdout:Generating a dashboard self-signed certificate... 2024-03-12T03:09:05.800 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-12T03:09:05.800 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.912+0000 7feff4a8f700 1 Processor -- start 2024-03-12T03:09:05.801 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.912+0000 7feff4a8f700 1 -- start start 2024-03-12T03:09:05.801 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.912+0000 7feff4a8f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feff0107ec0 con 0x7feff0107300 2024-03-12T03:09:05.801 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.913+0000 7fefee59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7feff0107300 0x7feff0107720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37764/0 (socket says 172.21.15.94:37764) 2024-03-12T03:09:05.801 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.913+0000 7fefee59c700 1 -- 172.21.15.94:0/3194378569 learned_addr learned my addr 172.21.15.94:0/3194378569 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:05.801 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.913+0000 7fefed59a700 1 -- 172.21.15.94:0/3194378569 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593840235 0 0) 0x7feff0107ec0 con 0x7feff0107300 2024-03-12T03:09:05.801 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.913+0000 7fefed59a700 1 -- 172.21.15.94:0/3194378569 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd80035b0 con 0x7feff0107300 2024-03-12T03:09:05.801 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.913+0000 7fefed59a700 1 -- 172.21.15.94:0/3194378569 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1763665075 0 0) 0x7fefd80035b0 con 0x7feff0107300 2024-03-12T03:09:05.801 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.913+0000 7fefed59a700 1 -- 172.21.15.94:0/3194378569 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff010b2e0 con 0x7feff0107300 2024-03-12T03:09:05.801 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.913+0000 7fefed59a700 1 -- 172.21.15.94:0/3194378569 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fefe0001580 con 0x7feff0107300 2024-03-12T03:09:05.801 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.913+0000 7fefed59a700 1 -- 172.21.15.94:0/3194378569 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fefe0004930 con 0x7feff0107300 2024-03-12T03:09:05.801 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.913+0000 7fefed59a700 1 -- 172.21.15.94:0/3194378569 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fefe0005a70 con 0x7feff0107300 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.914+0000 7feff4a8f700 1 -- 172.21.15.94:0/3194378569 >> v1:172.21.15.94:6789/0 conn(0x7feff0107300 legacy=0x7feff0107720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.914+0000 7feff4a8f700 1 -- 172.21.15.94:0/3194378569 shutdown_connections 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.914+0000 7feff4a8f700 1 -- 172.21.15.94:0/3194378569 >> 172.21.15.94:0/3194378569 conn(0x7feff00ff8e0 msgr2=0x7feff0101d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.914+0000 7feff4a8f700 1 -- 172.21.15.94:0/3194378569 shutdown_connections 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.914+0000 7feff4a8f700 1 -- 172.21.15.94:0/3194378569 wait complete. 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.915+0000 7feff4a8f700 1 Processor -- start 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.915+0000 7feff4a8f700 1 -- start start 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.915+0000 7feff4a8f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feff019f8a0 con 0x7feff0107300 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.915+0000 7fefee59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7feff0107300 0x7feff0106920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37770/0 (socket says 172.21.15.94:37770) 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.915+0000 7fefee59c700 1 -- 172.21.15.94:0/614397128 learned_addr learned my addr 172.21.15.94:0/614397128 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.916+0000 7fefdf7fe700 1 -- 172.21.15.94:0/614397128 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979231946 0 0) 0x7feff019f8a0 con 0x7feff0107300 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.916+0000 7fefdf7fe700 1 -- 172.21.15.94:0/614397128 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd8005530 con 0x7feff0107300 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.916+0000 7fefdf7fe700 1 -- 172.21.15.94:0/614397128 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 92327168 0 0) 0x7fefd8005530 con 0x7feff0107300 2024-03-12T03:09:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.916+0000 7fefdf7fe700 1 -- 172.21.15.94:0/614397128 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feff019f8a0 con 0x7feff0107300 2024-03-12T03:09:05.803 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.916+0000 7fefdf7fe700 1 -- 172.21.15.94:0/614397128 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fefe0001080 con 0x7feff0107300 2024-03-12T03:09:05.803 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.916+0000 7fefdf7fe700 1 -- 172.21.15.94:0/614397128 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3121666550 0 0) 0x7feff019f8a0 con 0x7feff0107300 2024-03-12T03:09:05.803 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.916+0000 7fefdf7fe700 1 -- 172.21.15.94:0/614397128 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff01a0a80 con 0x7feff0107300 2024-03-12T03:09:05.803 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.916+0000 7feff4a8f700 1 -- 172.21.15.94:0/614397128 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff019fad0 con 0x7feff0107300 2024-03-12T03:09:05.803 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.916+0000 7feff4a8f700 1 -- 172.21.15.94:0/614397128 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feff01a0010 con 0x7feff0107300 2024-03-12T03:09:05.803 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.916+0000 7fefdf7fe700 1 -- 172.21.15.94:0/614397128 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fefe0005a60 con 0x7feff0107300 2024-03-12T03:09:05.803 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.916+0000 7fefdf7fe700 1 -- 172.21.15.94:0/614397128 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fefe00062c0 con 0x7feff0107300 2024-03-12T03:09:05.803 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.917+0000 7fefdf7fe700 1 -- 172.21.15.94:0/614397128 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 12) v1 ==== 46084+0+0 (unknown 3538525251 0 0) 0x7fefe0011930 con 0x7feff0107300 2024-03-12T03:09:05.803 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.917+0000 7feff4a8f700 1 -- 172.21.15.94:0/614397128 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd0005150 con 0x7feff0107300 2024-03-12T03:09:05.803 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.918+0000 7fefdf7fe700 1 -- 172.21.15.94:0/614397128 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 2889753012 0 0) 0x7fefe0048870 con 0x7feff0107300 2024-03-12T03:09:05.803 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:04.923+0000 7fefdf7fe700 1 -- 172.21.15.94:0/614397128 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fefe00178a0 con 0x7feff0107300 2024-03-12T03:09:05.803 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:05.131+0000 7feff4a8f700 1 -- 172.21.15.94:0/614397128 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fefd0000bc0 con 0x7fefd803a410 2024-03-12T03:09:05.803 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:05.210+0000 7fefdf7fe700 1 -- 172.21.15.94:0/614397128 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (unknown 0 0 3317752739) 0x7fefd0000bc0 con 0x7fefd803a410 2024-03-12T03:09:05.804 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:05.212+0000 7feff4a8f700 1 -- 172.21.15.94:0/614397128 >> v1:172.21.15.94:6800/1299282424 conn(0x7fefd803a410 legacy=0x7fefd803c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:05.804 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:05.212+0000 7feff4a8f700 1 -- 172.21.15.94:0/614397128 >> v1:172.21.15.94:6789/0 conn(0x7feff0107300 legacy=0x7feff0106920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:05.804 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:05.212+0000 7feff4a8f700 1 -- 172.21.15.94:0/614397128 shutdown_connections 2024-03-12T03:09:05.804 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:05.212+0000 7feff4a8f700 1 -- 172.21.15.94:0/614397128 >> 172.21.15.94:0/614397128 conn(0x7feff00ff8e0 msgr2=0x7feff0101d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:05.804 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:05.213+0000 7feff4a8f700 1 -- 172.21.15.94:0/614397128 shutdown_connections 2024-03-12T03:09:05.804 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:05.213+0000 7feff4a8f700 1 -- 172.21.15.94:0/614397128 wait complete. 2024-03-12T03:09:05.804 INFO:teuthology.orchestra.run.smithi094.stdout:Creating initial admin user... 2024-03-12T03:09:06.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:05 smithi094 ceph-mon[29291]: mgrmap e12: y(active, since 2s) 2024-03-12T03:09:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:05 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:05 smithi094 ceph-mon[29291]: from='client.14164 v1:172.21.15.94:0/614397128' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:09:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:05 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:05 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:07.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:06 smithi094 ceph-mon[29291]: from='client.14166 v1:172.21.15.94:0/3656285751' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:09:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:07.253 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$5e2yFj.Q4gF1CF2Z4oaXf.gTY4s.94C2Qlr3G5FjzGtO7Z6nlHrVi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710212946, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-12T03:09:07.253 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.232+0000 7f95ca651700 1 Processor -- start 2024-03-12T03:09:07.253 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.232+0000 7f95ca651700 1 -- start start 2024-03-12T03:09:07.253 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.232+0000 7f95ca651700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f95c410a1d0 con 0x7f95c4104010 2024-03-12T03:09:07.253 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.232+0000 7f95c3fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f95c4104010 0x7f95c4104430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37772/0 (socket says 172.21.15.94:37772) 2024-03-12T03:09:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.232+0000 7f95c3fff700 1 -- 172.21.15.94:0/3497988347 learned_addr learned my addr 172.21.15.94:0/3497988347 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.232+0000 7f95c2ffd700 1 -- 172.21.15.94:0/3497988347 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460556030 0 0) 0x7f95c410a1d0 con 0x7f95c4104010 2024-03-12T03:09:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.232+0000 7f95c2ffd700 1 -- 172.21.15.94:0/3497988347 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ac0035b0 con 0x7f95c4104010 2024-03-12T03:09:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.232+0000 7f95c2ffd700 1 -- 172.21.15.94:0/3497988347 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 892580951 0 0) 0x7f95ac0035b0 con 0x7f95c4104010 2024-03-12T03:09:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.233+0000 7f95c2ffd700 1 -- 172.21.15.94:0/3497988347 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c410b3b0 con 0x7f95c4104010 2024-03-12T03:09:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.233+0000 7f95c2ffd700 1 -- 172.21.15.94:0/3497988347 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f95b8001580 con 0x7f95c4104010 2024-03-12T03:09:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.233+0000 7f95c2ffd700 1 -- 172.21.15.94:0/3497988347 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f95b8004930 con 0x7f95c4104010 2024-03-12T03:09:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.233+0000 7f95c2ffd700 1 -- 172.21.15.94:0/3497988347 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f95b8005a70 con 0x7f95c4104010 2024-03-12T03:09:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.233+0000 7f95ca651700 1 -- 172.21.15.94:0/3497988347 >> v1:172.21.15.94:6789/0 conn(0x7f95c4104010 legacy=0x7f95c4104430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.233+0000 7f95ca651700 1 -- 172.21.15.94:0/3497988347 shutdown_connections 2024-03-12T03:09:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.233+0000 7f95ca651700 1 -- 172.21.15.94:0/3497988347 >> 172.21.15.94:0/3497988347 conn(0x7f95c40ff920 msgr2=0x7f95c4101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.233+0000 7f95ca651700 1 -- 172.21.15.94:0/3497988347 shutdown_connections 2024-03-12T03:09:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.233+0000 7f95ca651700 1 -- 172.21.15.94:0/3497988347 wait complete. 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.234+0000 7f95ca651700 1 Processor -- start 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.234+0000 7f95ca651700 1 -- start start 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.234+0000 7f95ca651700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f95c4199f60 con 0x7f95c4104010 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.234+0000 7f95c3fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f95c4104010 0x7f95c4199850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37788/0 (socket says 172.21.15.94:37788) 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.234+0000 7f95c3fff700 1 -- 172.21.15.94:0/3656285751 learned_addr learned my addr 172.21.15.94:0/3656285751 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.234+0000 7f95c17fa700 1 -- 172.21.15.94:0/3656285751 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384788689 0 0) 0x7f95c4199f60 con 0x7f95c4104010 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.234+0000 7f95c17fa700 1 -- 172.21.15.94:0/3656285751 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ac005530 con 0x7f95c4104010 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.235+0000 7f95c17fa700 1 -- 172.21.15.94:0/3656285751 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 123410457 0 0) 0x7f95ac005530 con 0x7f95c4104010 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.235+0000 7f95c17fa700 1 -- 172.21.15.94:0/3656285751 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95c4199f60 con 0x7f95c4104010 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.235+0000 7f95c17fa700 1 -- 172.21.15.94:0/3656285751 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f95b8001080 con 0x7f95c4104010 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.235+0000 7f95c17fa700 1 -- 172.21.15.94:0/3656285751 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3277722240 0 0) 0x7f95c4199f60 con 0x7f95c4104010 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.235+0000 7f95c17fa700 1 -- 172.21.15.94:0/3656285751 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c419a130 con 0x7f95c4104010 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.235+0000 7f95ca651700 1 -- 172.21.15.94:0/3656285751 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c419a420 con 0x7f95c4104010 2024-03-12T03:09:07.255 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.235+0000 7f95ca651700 1 -- 172.21.15.94:0/3656285751 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c419df70 con 0x7f95c4104010 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.235+0000 7f95c17fa700 1 -- 172.21.15.94:0/3656285751 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f95b8005a60 con 0x7f95c4104010 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.235+0000 7f95c17fa700 1 -- 172.21.15.94:0/3656285751 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f95b80063c0 con 0x7f95c4104010 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.236+0000 7f95c17fa700 1 -- 172.21.15.94:0/3656285751 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 12) v1 ==== 46084+0+0 (unknown 3538525251 0 0) 0x7f95b8011990 con 0x7f95c4104010 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.236+0000 7f95ca651700 1 -- 172.21.15.94:0/3656285751 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b0005150 con 0x7f95c4104010 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.236+0000 7f95c17fa700 1 -- 172.21.15.94:0/3656285751 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 2889753012 0 0) 0x7f95b80478d0 con 0x7f95c4104010 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.241+0000 7f95c17fa700 1 -- 172.21.15.94:0/3656285751 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f95b8005d00 con 0x7f95c4104010 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.458+0000 7f95ca651700 1 -- 172.21.15.94:0/3656285751 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f95b0002430 con 0x7f95ac03a460 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.679+0000 7f95c17fa700 1 -- 172.21.15.94:0/3656285751 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (unknown 0 0 3597097711) 0x7f95b0002430 con 0x7f95ac03a460 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.681+0000 7f95ca651700 1 -- 172.21.15.94:0/3656285751 >> v1:172.21.15.94:6800/1299282424 conn(0x7f95ac03a460 legacy=0x7f95ac03c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.681+0000 7f95ca651700 1 -- 172.21.15.94:0/3656285751 >> v1:172.21.15.94:6789/0 conn(0x7f95c4104010 legacy=0x7f95c4199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.681+0000 7f95ca651700 1 -- 172.21.15.94:0/3656285751 shutdown_connections 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.681+0000 7f95ca651700 1 -- 172.21.15.94:0/3656285751 >> 172.21.15.94:0/3656285751 conn(0x7f95c40ff920 msgr2=0x7f95c4101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.681+0000 7f95ca651700 1 -- 172.21.15.94:0/3656285751 shutdown_connections 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:06.681+0000 7f95ca651700 1 -- 172.21.15.94:0/3656285751 wait complete. 2024-03-12T03:09:07.256 INFO:teuthology.orchestra.run.smithi094.stdout:Fetching dashboard port number... 2024-03-12T03:09:08.122 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2457745700' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-12T03:09:09.094 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 8443 2024-03-12T03:09:09.095 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.706+0000 7f6fe2594700 1 Processor -- start 2024-03-12T03:09:09.095 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.706+0000 7f6fe2594700 1 -- start start 2024-03-12T03:09:09.095 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.706+0000 7f6fe2594700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6fdc1044f0 con 0x7f6fdc103930 2024-03-12T03:09:09.095 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.706+0000 7f6fdbfff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6fdc103930 0x7f6fdc103d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37792/0 (socket says 172.21.15.94:37792) 2024-03-12T03:09:09.095 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.706+0000 7f6fdbfff700 1 -- 172.21.15.94:0/3285027697 learned_addr learned my addr 172.21.15.94:0/3285027697 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:09.095 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.707+0000 7f6fdaffd700 1 -- 172.21.15.94:0/3285027697 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445907188 0 0) 0x7f6fdc1044f0 con 0x7f6fdc103930 2024-03-12T03:09:09.095 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.707+0000 7f6fdaffd700 1 -- 172.21.15.94:0/3285027697 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc40035b0 con 0x7f6fdc103930 2024-03-12T03:09:09.095 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.707+0000 7f6fdaffd700 1 -- 172.21.15.94:0/3285027697 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3142663281 0 0) 0x7f6fc40035b0 con 0x7f6fdc103930 2024-03-12T03:09:09.095 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.707+0000 7f6fdaffd700 1 -- 172.21.15.94:0/3285027697 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc1046c0 con 0x7f6fdc103930 2024-03-12T03:09:09.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.707+0000 7f6fdaffd700 1 -- 172.21.15.94:0/3285027697 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f6fcc001580 con 0x7f6fdc103930 2024-03-12T03:09:09.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.707+0000 7f6fdaffd700 1 -- 172.21.15.94:0/3285027697 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f6fcc004930 con 0x7f6fdc103930 2024-03-12T03:09:09.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.707+0000 7f6fdaffd700 1 -- 172.21.15.94:0/3285027697 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f6fcc005a70 con 0x7f6fdc103930 2024-03-12T03:09:09.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.707+0000 7f6fe2594700 1 -- 172.21.15.94:0/3285027697 >> v1:172.21.15.94:6789/0 conn(0x7f6fdc103930 legacy=0x7f6fdc103d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:09.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.708+0000 7f6fe2594700 1 -- 172.21.15.94:0/3285027697 shutdown_connections 2024-03-12T03:09:09.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.708+0000 7f6fe2594700 1 -- 172.21.15.94:0/3285027697 >> 172.21.15.94:0/3285027697 conn(0x7f6fdc0ff480 msgr2=0x7f6fdc1018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:09.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.708+0000 7f6fe2594700 1 -- 172.21.15.94:0/3285027697 shutdown_connections 2024-03-12T03:09:09.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.708+0000 7f6fe2594700 1 -- 172.21.15.94:0/3285027697 wait complete. 2024-03-12T03:09:09.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.708+0000 7f6fe2594700 1 Processor -- start 2024-03-12T03:09:09.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.708+0000 7f6fe2594700 1 -- start start 2024-03-12T03:09:09.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.708+0000 7f6fe2594700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6fdc19f870 con 0x7f6fdc103930 2024-03-12T03:09:09.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.709+0000 7f6fdbfff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6fdc103930 0x7f6fdc106cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37796/0 (socket says 172.21.15.94:37796) 2024-03-12T03:09:09.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.709+0000 7f6fdbfff700 1 -- 172.21.15.94:0/2457745700 learned_addr learned my addr 172.21.15.94:0/2457745700 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:09.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.709+0000 7f6fd97fa700 1 -- 172.21.15.94:0/2457745700 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707979346 0 0) 0x7f6fdc19f870 con 0x7f6fdc103930 2024-03-12T03:09:09.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.709+0000 7f6fd97fa700 1 -- 172.21.15.94:0/2457745700 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc4005530 con 0x7f6fdc103930 2024-03-12T03:09:09.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.709+0000 7f6fd97fa700 1 -- 172.21.15.94:0/2457745700 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3602711781 0 0) 0x7f6fc4005530 con 0x7f6fdc103930 2024-03-12T03:09:09.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.709+0000 7f6fd97fa700 1 -- 172.21.15.94:0/2457745700 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fdc19f870 con 0x7f6fdc103930 2024-03-12T03:09:09.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.709+0000 7f6fd97fa700 1 -- 172.21.15.94:0/2457745700 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f6fcc001080 con 0x7f6fdc103930 2024-03-12T03:09:09.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.709+0000 7f6fd97fa700 1 -- 172.21.15.94:0/2457745700 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3274094429 0 0) 0x7f6fdc19f870 con 0x7f6fdc103930 2024-03-12T03:09:09.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.709+0000 7f6fd97fa700 1 -- 172.21.15.94:0/2457745700 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc1a0a50 con 0x7f6fdc103930 2024-03-12T03:09:09.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.709+0000 7f6fe2594700 1 -- 172.21.15.94:0/2457745700 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fdc19faa0 con 0x7f6fdc103930 2024-03-12T03:09:09.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.709+0000 7f6fe2594700 1 -- 172.21.15.94:0/2457745700 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fdc19ffe0 con 0x7f6fdc103930 2024-03-12T03:09:09.098 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.709+0000 7f6fd97fa700 1 -- 172.21.15.94:0/2457745700 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f6fcc004930 con 0x7f6fdc103930 2024-03-12T03:09:09.098 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.709+0000 7f6fd97fa700 1 -- 172.21.15.94:0/2457745700 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f6fcc006130 con 0x7f6fdc103930 2024-03-12T03:09:09.098 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.710+0000 7f6fd97fa700 1 -- 172.21.15.94:0/2457745700 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 12) v1 ==== 46084+0+0 (unknown 3538525251 0 0) 0x7f6fcc011720 con 0x7f6fdc103930 2024-03-12T03:09:09.098 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.710+0000 7f6fe2594700 1 -- 172.21.15.94:0/2457745700 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc8005150 con 0x7f6fdc103930 2024-03-12T03:09:09.098 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.710+0000 7f6fd97fa700 1 -- 172.21.15.94:0/2457745700 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 2889753012 0 0) 0x7f6fcc047940 con 0x7f6fdc103930 2024-03-12T03:09:09.098 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.715+0000 7f6fd97fa700 1 -- 172.21.15.94:0/2457745700 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f6fcc0176c0 con 0x7f6fdc103930 2024-03-12T03:09:09.098 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.913+0000 7f6fe2594700 1 -- 172.21.15.94:0/2457745700 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f6fc8005440 con 0x7f6fdc103930 2024-03-12T03:09:09.098 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.914+0000 7f6fd97fa700 1 -- 172.21.15.94:0/2457745700 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (unknown 3713421687 0 83753974) 0x7f6fcc016fc0 con 0x7f6fdc103930 2024-03-12T03:09:09.098 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.916+0000 7f6fe2594700 1 -- 172.21.15.94:0/2457745700 >> v1:172.21.15.94:6800/1299282424 conn(0x7f6fc403a3c0 legacy=0x7f6fc403c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:09.098 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.916+0000 7f6fe2594700 1 -- 172.21.15.94:0/2457745700 >> v1:172.21.15.94:6789/0 conn(0x7f6fdc103930 legacy=0x7f6fdc106cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:09.098 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.916+0000 7f6fe2594700 1 -- 172.21.15.94:0/2457745700 shutdown_connections 2024-03-12T03:09:09.099 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.916+0000 7f6fe2594700 1 -- 172.21.15.94:0/2457745700 >> 172.21.15.94:0/2457745700 conn(0x7f6fdc0ff480 msgr2=0x7f6fdc107af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:09.099 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.916+0000 7f6fe2594700 1 -- 172.21.15.94:0/2457745700 shutdown_connections 2024-03-12T03:09:09.099 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:07.916+0000 7f6fe2594700 1 -- 172.21.15.94:0/2457745700 wait complete. 2024-03-12T03:09:09.102 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:09:09.102 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout disabled 2024-03-12T03:09:09.110 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:09:09.110 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-03-12T03:09:09.110 INFO:teuthology.orchestra.run.smithi094.stdout:firewalld.service is not enabled 2024-03-12T03:09:09.111 INFO:teuthology.orchestra.run.smithi094.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-12T03:09:09.113 INFO:teuthology.orchestra.run.smithi094.stdout:Ceph Dashboard is now available at: 2024-03-12T03:09:09.113 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:09:09.113 INFO:teuthology.orchestra.run.smithi094.stdout: URL: https://smithi094.front.sepia.ceph.com:8443/ 2024-03-12T03:09:09.113 INFO:teuthology.orchestra.run.smithi094.stdout: User: admin 2024-03-12T03:09:09.113 INFO:teuthology.orchestra.run.smithi094.stdout: Password: ltkxkhggzh 2024-03-12T03:09:09.113 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:09:09.113 INFO:teuthology.orchestra.run.smithi094.stdout:Saving cluster configuration to /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config directory 2024-03-12T03:09:09.114 INFO:teuthology.orchestra.run.smithi094.stdout:Enabling autotune for osd_memory_target 2024-03-12T03:09:10.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:09 smithi094 ceph-mon[29291]: mgrmap e13: y(active, since 6s) 2024-03-12T03:09:10.434 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.578+0000 7f988da9f700 1 Processor -- start 2024-03-12T03:09:10.434 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.578+0000 7f988da9f700 1 -- start start 2024-03-12T03:09:10.435 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.578+0000 7f988da9f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f98880795f0 con 0x7f9888106840 2024-03-12T03:09:10.435 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.578+0000 7f98877fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9888106840 0x7f9888108c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37806/0 (socket says 172.21.15.94:37806) 2024-03-12T03:09:10.435 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.578+0000 7f98877fe700 1 -- 172.21.15.94:0/2428940478 learned_addr learned my addr 172.21.15.94:0/2428940478 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:10.435 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.579+0000 7f98867fc700 1 -- 172.21.15.94:0/2428940478 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618352120 0 0) 0x7f98880795f0 con 0x7f9888106840 2024-03-12T03:09:10.435 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.579+0000 7f98867fc700 1 -- 172.21.15.94:0/2428940478 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98700035b0 con 0x7f9888106840 2024-03-12T03:09:10.435 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.579+0000 7f98867fc700 1 -- 172.21.15.94:0/2428940478 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4089710046 0 0) 0x7f98700035b0 con 0x7f9888106840 2024-03-12T03:09:10.435 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.579+0000 7f98867fc700 1 -- 172.21.15.94:0/2428940478 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988810b360 con 0x7f9888106840 2024-03-12T03:09:10.435 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.579+0000 7f98867fc700 1 -- 172.21.15.94:0/2428940478 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f9878001580 con 0x7f9888106840 2024-03-12T03:09:10.435 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.579+0000 7f98867fc700 1 -- 172.21.15.94:0/2428940478 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f9878004930 con 0x7f9888106840 2024-03-12T03:09:10.435 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.579+0000 7f98867fc700 1 -- 172.21.15.94:0/2428940478 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f9878005a70 con 0x7f9888106840 2024-03-12T03:09:10.435 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.580+0000 7f988da9f700 1 -- 172.21.15.94:0/2428940478 >> v1:172.21.15.94:6789/0 conn(0x7f9888106840 legacy=0x7f9888108c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:10.435 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.580+0000 7f988da9f700 1 -- 172.21.15.94:0/2428940478 shutdown_connections 2024-03-12T03:09:10.435 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.580+0000 7f988da9f700 1 -- 172.21.15.94:0/2428940478 >> 172.21.15.94:0/2428940478 conn(0x7f98881001b0 msgr2=0x7f98881025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:10.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.580+0000 7f988da9f700 1 -- 172.21.15.94:0/2428940478 shutdown_connections 2024-03-12T03:09:10.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.580+0000 7f988da9f700 1 -- 172.21.15.94:0/2428940478 wait complete. 2024-03-12T03:09:10.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.580+0000 7f988da9f700 1 Processor -- start 2024-03-12T03:09:10.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.580+0000 7f988da9f700 1 -- start start 2024-03-12T03:09:10.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f988da9f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9888199f80 con 0x7f9888106840 2024-03-12T03:09:10.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f98877fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9888106840 0x7f9888199870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37812/0 (socket says 172.21.15.94:37812) 2024-03-12T03:09:10.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f98877fe700 1 -- 172.21.15.94:0/1512584575 learned_addr learned my addr 172.21.15.94:0/1512584575 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:10.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f9884ff9700 1 -- 172.21.15.94:0/1512584575 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500177576 0 0) 0x7f9888199f80 con 0x7f9888106840 2024-03-12T03:09:10.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f9884ff9700 1 -- 172.21.15.94:0/1512584575 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9870005530 con 0x7f9888106840 2024-03-12T03:09:10.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f9884ff9700 1 -- 172.21.15.94:0/1512584575 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3809056002 0 0) 0x7f9870005530 con 0x7f9888106840 2024-03-12T03:09:10.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f9884ff9700 1 -- 172.21.15.94:0/1512584575 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9888199f80 con 0x7f9888106840 2024-03-12T03:09:10.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f9884ff9700 1 -- 172.21.15.94:0/1512584575 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f9878001080 con 0x7f9888106840 2024-03-12T03:09:10.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f9884ff9700 1 -- 172.21.15.94:0/1512584575 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3398399979 0 0) 0x7f9888199f80 con 0x7f9888106840 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f9884ff9700 1 -- 172.21.15.94:0/1512584575 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988819a150 con 0x7f9888106840 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f988da9f700 1 -- 172.21.15.94:0/1512584575 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988819a440 con 0x7f9888106840 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f988da9f700 1 -- 172.21.15.94:0/1512584575 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f988819df90 con 0x7f9888106840 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f9884ff9700 1 -- 172.21.15.94:0/1512584575 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f9878005110 con 0x7f9888106840 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.581+0000 7f9884ff9700 1 -- 172.21.15.94:0/1512584575 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f98780061a0 con 0x7f9888106840 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.582+0000 7f9884ff9700 1 -- 172.21.15.94:0/1512584575 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7f9878011780 con 0x7f9888106840 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.582+0000 7f988da9f700 1 -- 172.21.15.94:0/1512584575 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9874005120 con 0x7f9888106840 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.582+0000 7f9884ff9700 1 -- 172.21.15.94:0/1512584575 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 2889753012 0 0) 0x7f9878047a70 con 0x7f9888106840 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.588+0000 7f9884ff9700 1 -- 172.21.15.94:0/1512584575 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f9878017790 con 0x7f9888106840 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.783+0000 7f988da9f700 1 -- 172.21.15.94:0/1512584575 --> v1:172.21.15.94:6789/0 -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f9874005410 con 0x7f9888106840 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.784+0000 7f9884ff9700 1 -- 172.21.15.94:0/1512584575 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (unknown 689008156 0 0) 0x7f9878017090 con 0x7f9888106840 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.786+0000 7f988da9f700 1 -- 172.21.15.94:0/1512584575 >> v1:172.21.15.94:6800/1299282424 conn(0x7f987003a4b0 legacy=0x7f987003c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.786+0000 7f988da9f700 1 -- 172.21.15.94:0/1512584575 >> v1:172.21.15.94:6789/0 conn(0x7f9888106840 legacy=0x7f9888199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.786+0000 7f988da9f700 1 -- 172.21.15.94:0/1512584575 shutdown_connections 2024-03-12T03:09:10.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.786+0000 7f988da9f700 1 -- 172.21.15.94:0/1512584575 >> 172.21.15.94:0/1512584575 conn(0x7f98881001b0 msgr2=0x7f98881904e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:10.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.786+0000 7f988da9f700 1 -- 172.21.15.94:0/1512584575 shutdown_connections 2024-03-12T03:09:10.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:09.786+0000 7f988da9f700 1 -- 172.21.15.94:0/1512584575 wait complete. 2024-03-12T03:09:11.784 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.872+0000 7fa8fd126700 1 Processor -- start 2024-03-12T03:09:11.784 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.872+0000 7fa8fd126700 1 -- start start 2024-03-12T03:09:11.784 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.872+0000 7fa8fd126700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa8f8109030 con 0x7fa8f8104c00 2024-03-12T03:09:11.784 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.872+0000 7fa8f6d9d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa8f8104c00 0x7fa8f8105020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37824/0 (socket says 172.21.15.94:37824) 2024-03-12T03:09:11.784 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.872+0000 7fa8f6d9d700 1 -- 172.21.15.94:0/2005075884 learned_addr learned my addr 172.21.15.94:0/2005075884 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:11.784 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.872+0000 7fa8f5d9b700 1 -- 172.21.15.94:0/2005075884 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230064663 0 0) 0x7fa8f8109030 con 0x7fa8f8104c00 2024-03-12T03:09:11.784 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.872+0000 7fa8f5d9b700 1 -- 172.21.15.94:0/2005075884 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e00035b0 con 0x7fa8f8104c00 2024-03-12T03:09:11.784 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.872+0000 7fa8f5d9b700 1 -- 172.21.15.94:0/2005075884 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2658025482 0 0) 0x7fa8e00035b0 con 0x7fa8f8104c00 2024-03-12T03:09:11.784 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.872+0000 7fa8f5d9b700 1 -- 172.21.15.94:0/2005075884 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f810a210 con 0x7fa8f8104c00 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.872+0000 7fa8f5d9b700 1 -- 172.21.15.94:0/2005075884 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fa8ec001580 con 0x7fa8f8104c00 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.872+0000 7fa8f5d9b700 1 -- 172.21.15.94:0/2005075884 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fa8ec001c70 con 0x7fa8f8104c00 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.872+0000 7fa8f5d9b700 1 -- 172.21.15.94:0/2005075884 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fa8ec005960 con 0x7fa8f8104c00 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.873+0000 7fa8fd126700 1 -- 172.21.15.94:0/2005075884 >> v1:172.21.15.94:6789/0 conn(0x7fa8f8104c00 legacy=0x7fa8f8105020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.873+0000 7fa8fd126700 1 -- 172.21.15.94:0/2005075884 shutdown_connections 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.873+0000 7fa8fd126700 1 -- 172.21.15.94:0/2005075884 >> 172.21.15.94:0/2005075884 conn(0x7fa8f8100160 msgr2=0x7fa8f81025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.873+0000 7fa8fd126700 1 -- 172.21.15.94:0/2005075884 shutdown_connections 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.873+0000 7fa8fd126700 1 -- 172.21.15.94:0/2005075884 wait complete. 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.874+0000 7fa8fd126700 1 Processor -- start 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.874+0000 7fa8fd126700 1 -- start start 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.874+0000 7fa8fd126700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa8f8198e70 con 0x7fa8f8104c00 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.874+0000 7fa8f6d9d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa8f8104c00 0x7fa8f8198760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37832/0 (socket says 172.21.15.94:37832) 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.874+0000 7fa8f6d9d700 1 -- 172.21.15.94:0/3846298718 learned_addr learned my addr 172.21.15.94:0/3846298718 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.874+0000 7fa8dffff700 1 -- 172.21.15.94:0/3846298718 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450584395 0 0) 0x7fa8f8198e70 con 0x7fa8f8104c00 2024-03-12T03:09:11.785 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.874+0000 7fa8dffff700 1 -- 172.21.15.94:0/3846298718 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e0005580 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.874+0000 7fa8dffff700 1 -- 172.21.15.94:0/3846298718 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3607851371 0 0) 0x7fa8e0005580 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.874+0000 7fa8dffff700 1 -- 172.21.15.94:0/3846298718 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8f8198e70 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.874+0000 7fa8dffff700 1 -- 172.21.15.94:0/3846298718 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fa8ec001080 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.875+0000 7fa8dffff700 1 -- 172.21.15.94:0/3846298718 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 127399637 0 0) 0x7fa8f8198e70 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.875+0000 7fa8dffff700 1 -- 172.21.15.94:0/3846298718 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f8199040 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.875+0000 7fa8fd126700 1 -- 172.21.15.94:0/3846298718 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f8199330 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.875+0000 7fa8fd126700 1 -- 172.21.15.94:0/3846298718 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f819ce80 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.875+0000 7fa8dffff700 1 -- 172.21.15.94:0/3846298718 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fa8ec0055c0 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.875+0000 7fa8dffff700 1 -- 172.21.15.94:0/3846298718 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fa8ec006130 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.876+0000 7fa8dffff700 1 -- 172.21.15.94:0/3846298718 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7fa8ec011710 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.876+0000 7fa8fd126700 1 -- 172.21.15.94:0/3846298718 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e4005150 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.876+0000 7fa8dffff700 1 -- 172.21.15.94:0/3846298718 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 2889753012 0 0) 0x7fa8ec0478f0 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:10.881+0000 7fa8dffff700 1 -- 172.21.15.94:0/3846298718 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fa8ec017720 con 0x7fa8f8104c00 2024-03-12T03:09:11.786 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:11.183+0000 7fa8fd126700 1 -- 172.21.15.94:0/3846298718 --> v1:172.21.15.94:6789/0 -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fa8e4005440 con 0x7fa8f8104c00 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:11.185+0000 7fa8dffff700 1 -- 172.21.15.94:0/3846298718 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (unknown 1169358022 0 0) 0x7fa8ec017020 con 0x7fa8f8104c00 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:11.187+0000 7fa8fd126700 1 -- 172.21.15.94:0/3846298718 >> v1:172.21.15.94:6800/1299282424 conn(0x7fa8e003a480 legacy=0x7fa8e003c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:11.187+0000 7fa8fd126700 1 -- 172.21.15.94:0/3846298718 >> v1:172.21.15.94:6789/0 conn(0x7fa8f8104c00 legacy=0x7fa8f8198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:11.187+0000 7fa8fd126700 1 -- 172.21.15.94:0/3846298718 shutdown_connections 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:11.187+0000 7fa8fd126700 1 -- 172.21.15.94:0/3846298718 >> 172.21.15.94:0/3846298718 conn(0x7fa8f8100160 msgr2=0x7fa8f81025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:11.187+0000 7fa8fd126700 1 -- 172.21.15.94:0/3846298718 shutdown_connections 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-03-12T03:09:11.187+0000 7fa8fd126700 1 -- 172.21.15.94:0/3846298718 wait complete. 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: sudo /sbin/cephadm shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout:Or, if you are only running a single cluster on this host: 2024-03-12T03:09:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:09:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: sudo /sbin/cephadm shell 2024-03-12T03:09:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:09:11.788 INFO:teuthology.orchestra.run.smithi094.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-12T03:09:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:09:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: ceph telemetry on 2024-03-12T03:09:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:09:11.788 INFO:teuthology.orchestra.run.smithi094.stdout:For more information see: 2024-03-12T03:09:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:09:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-12T03:09:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:09:11.788 INFO:teuthology.orchestra.run.smithi094.stdout:Bootstrap complete. 2024-03-12T03:09:11.823 INFO:tasks.cephadm:Fetching config... 2024-03-12T03:09:11.823 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:09:11.823 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-12T03:09:11.839 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-12T03:09:11.839 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:09:11.839 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-12T03:09:11.896 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-12T03:09:11.896 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:09:11.897 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-12T03:09:11.964 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-12T03:09:11.964 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:09:11.964 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-12T03:09:12.019 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-12T03:09:12.020 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiwCWw9OXTQjngODjhbgtVKcfxbxEg/jlDJd4myYgBMFKsXT4DYeh0rxng9JViU9MlZJbgFOil0gNMEvcS3bVJPJFkGYdOeaXcAWT6Z2r7ohukEib5AY/Pr9+SpMBwyh+pEmt/2npEC3Nap300wGy3krPs4uoFv8MZbOuvjIW2h5CgxMM7k9jiArk4B69fzzk9E1r/jCxQKAXuRJsWN8fR5BLF13g97cQJmuotj8kGRrDykwnW9X7aBOV7AFuOvcCP4JSSyzydVrMHAFcGEOYoA1Az3MV+xwAKuxcxd41xvoQXaJOqRIGSPJCDfPWPGgiYmG38r/lTCYAtGMmXJbhfflny1spaeCefNnsIVU43Z8abyjDyK6xW1anZaqD2uVbortx+A6NYmn6caW1ak8VQUgvtb6VjtEYjw3Lbp0DtVFbvS2mJp3nj/k4GGSdkIMLH01LfMKAi8oFoviGqCUsX9FdNqxxr1HubRwbOR20UQQrzVtAIx50Oc4Zwie2Ma60= ceph-964142e8-e01d-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T03:09:12.097 INFO:teuthology.orchestra.run.smithi094.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiwCWw9OXTQjngODjhbgtVKcfxbxEg/jlDJd4myYgBMFKsXT4DYeh0rxng9JViU9MlZJbgFOil0gNMEvcS3bVJPJFkGYdOeaXcAWT6Z2r7ohukEib5AY/Pr9+SpMBwyh+pEmt/2npEC3Nap300wGy3krPs4uoFv8MZbOuvjIW2h5CgxMM7k9jiArk4B69fzzk9E1r/jCxQKAXuRJsWN8fR5BLF13g97cQJmuotj8kGRrDykwnW9X7aBOV7AFuOvcCP4JSSyzydVrMHAFcGEOYoA1Az3MV+xwAKuxcxd41xvoQXaJOqRIGSPJCDfPWPGgiYmG38r/lTCYAtGMmXJbhfflny1spaeCefNnsIVU43Z8abyjDyK6xW1anZaqD2uVbortx+A6NYmn6caW1ak8VQUgvtb6VjtEYjw3Lbp0DtVFbvS2mJp3nj/k4GGSdkIMLH01LfMKAi8oFoviGqCUsX9FdNqxxr1HubRwbOR20UQQrzVtAIx50Oc4Zwie2Ma60= ceph-964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:09:12.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiwCWw9OXTQjngODjhbgtVKcfxbxEg/jlDJd4myYgBMFKsXT4DYeh0rxng9JViU9MlZJbgFOil0gNMEvcS3bVJPJFkGYdOeaXcAWT6Z2r7ohukEib5AY/Pr9+SpMBwyh+pEmt/2npEC3Nap300wGy3krPs4uoFv8MZbOuvjIW2h5CgxMM7k9jiArk4B69fzzk9E1r/jCxQKAXuRJsWN8fR5BLF13g97cQJmuotj8kGRrDykwnW9X7aBOV7AFuOvcCP4JSSyzydVrMHAFcGEOYoA1Az3MV+xwAKuxcxd41xvoQXaJOqRIGSPJCDfPWPGgiYmG38r/lTCYAtGMmXJbhfflny1spaeCefNnsIVU43Z8abyjDyK6xW1anZaqD2uVbortx+A6NYmn6caW1ak8VQUgvtb6VjtEYjw3Lbp0DtVFbvS2mJp3nj/k4GGSdkIMLH01LfMKAi8oFoviGqCUsX9FdNqxxr1HubRwbOR20UQQrzVtAIx50Oc4Zwie2Ma60= ceph-964142e8-e01d-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T03:09:12.157 INFO:teuthology.orchestra.run.smithi151.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiwCWw9OXTQjngODjhbgtVKcfxbxEg/jlDJd4myYgBMFKsXT4DYeh0rxng9JViU9MlZJbgFOil0gNMEvcS3bVJPJFkGYdOeaXcAWT6Z2r7ohukEib5AY/Pr9+SpMBwyh+pEmt/2npEC3Nap300wGy3krPs4uoFv8MZbOuvjIW2h5CgxMM7k9jiArk4B69fzzk9E1r/jCxQKAXuRJsWN8fR5BLF13g97cQJmuotj8kGRrDykwnW9X7aBOV7AFuOvcCP4JSSyzydVrMHAFcGEOYoA1Az3MV+xwAKuxcxd41xvoQXaJOqRIGSPJCDfPWPGgiYmG38r/lTCYAtGMmXJbhfflny1spaeCefNnsIVU43Z8abyjDyK6xW1anZaqD2uVbortx+A6NYmn6caW1ak8VQUgvtb6VjtEYjw3Lbp0DtVFbvS2mJp3nj/k4GGSdkIMLH01LfMKAi8oFoviGqCUsX9FdNqxxr1HubRwbOR20UQQrzVtAIx50Oc4Zwie2Ma60= ceph-964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:09:12.172 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-12T03:09:12.290 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3846298718' entity='client.admin' 2024-03-12T03:09:12.490 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:09:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.224+0000 7f42243fc700 1 -- 172.21.15.94:0/3190210857 >> v1:172.21.15.94:6789/0 conn(0x7f421c101740 legacy=0x7f421c101b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.224+0000 7f42243fc700 1 -- 172.21.15.94:0/3190210857 shutdown_connections 2024-03-12T03:09:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.224+0000 7f42243fc700 1 -- 172.21.15.94:0/3190210857 >> 172.21.15.94:0/3190210857 conn(0x7f421c0fcce0 msgr2=0x7f421c0ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.224+0000 7f42243fc700 1 -- 172.21.15.94:0/3190210857 shutdown_connections 2024-03-12T03:09:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.224+0000 7f42243fc700 1 -- 172.21.15.94:0/3190210857 wait complete. 2024-03-12T03:09:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.224+0000 7f42243fc700 1 Processor -- start 2024-03-12T03:09:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.224+0000 7f42243fc700 1 -- start start 2024-03-12T03:09:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f42243fc700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f421c19c1e0 con 0x7f421c101740 2024-03-12T03:09:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f4222198700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f421c101740 0x7f421c075a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37854/0 (socket says 172.21.15.94:37854) 2024-03-12T03:09:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f4222198700 1 -- 172.21.15.94:0/2543352548 learned_addr learned my addr 172.21.15.94:0/2543352548 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f42137fe700 1 -- 172.21.15.94:0/2543352548 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750713129 0 0) 0x7f421c19c1e0 con 0x7f421c101740 2024-03-12T03:09:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f42137fe700 1 -- 172.21.15.94:0/2543352548 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4208005530 con 0x7f421c101740 2024-03-12T03:09:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f42137fe700 1 -- 172.21.15.94:0/2543352548 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3869204156 0 0) 0x7f4208005530 con 0x7f421c101740 2024-03-12T03:09:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f42137fe700 1 -- 172.21.15.94:0/2543352548 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f421c19c1e0 con 0x7f421c101740 2024-03-12T03:09:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f42137fe700 1 -- 172.21.15.94:0/2543352548 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f42140049e0 con 0x7f421c101740 2024-03-12T03:09:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f42137fe700 1 -- 172.21.15.94:0/2543352548 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1299805303 0 0) 0x7f421c19c1e0 con 0x7f421c101740 2024-03-12T03:09:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f42137fe700 1 -- 172.21.15.94:0/2543352548 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c19d3c0 con 0x7f421c101740 2024-03-12T03:09:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f42243fc700 1 -- 172.21.15.94:0/2543352548 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421c19c3b0 con 0x7f421c101740 2024-03-12T03:09:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f42243fc700 1 -- 172.21.15.94:0/2543352548 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f421c19c8f0 con 0x7f421c101740 2024-03-12T03:09:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f42137fe700 1 -- 172.21.15.94:0/2543352548 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f4214003ae0 con 0x7f421c101740 2024-03-12T03:09:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.225+0000 7f42137fe700 1 -- 172.21.15.94:0/2543352548 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f42140051a0 con 0x7f421c101740 2024-03-12T03:09:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.226+0000 7f42137fe700 1 -- 172.21.15.94:0/2543352548 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7f4214005360 con 0x7f421c101740 2024-03-12T03:09:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.226+0000 7f42137fe700 1 -- 172.21.15.94:0/2543352548 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 2889753012 0 0) 0x7f4214048950 con 0x7f421c101740 2024-03-12T03:09:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.226+0000 7f42243fc700 1 -- 172.21.15.94:0/2543352548 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4200005150 con 0x7f421c101740 2024-03-12T03:09:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.231+0000 7f42137fe700 1 -- 172.21.15.94:0/2543352548 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f42140178a0 con 0x7f421c101740 2024-03-12T03:09:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.427+0000 7f42243fc700 1 -- 172.21.15.94:0/2543352548 --> v1:172.21.15.94:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f4200005440 con 0x7f421c101740 2024-03-12T03:09:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.429+0000 7f42137fe700 1 -- 172.21.15.94:0/2543352548 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (unknown 3028693289 0 0) 0x7f42140171a0 con 0x7f421c101740 2024-03-12T03:09:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.431+0000 7f42243fc700 1 -- 172.21.15.94:0/2543352548 >> v1:172.21.15.94:6800/1299282424 conn(0x7f420803a2f0 legacy=0x7f420803c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.431+0000 7f42243fc700 1 -- 172.21.15.94:0/2543352548 >> v1:172.21.15.94:6789/0 conn(0x7f421c101740 legacy=0x7f421c075a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.431+0000 7f42243fc700 1 -- 172.21.15.94:0/2543352548 shutdown_connections 2024-03-12T03:09:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.432+0000 7f42243fc700 1 -- 172.21.15.94:0/2543352548 >> 172.21.15.94:0/2543352548 conn(0x7f421c0fcce0 msgr2=0x7f421c0ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.432+0000 7f42243fc700 1 -- 172.21.15.94:0/2543352548 shutdown_connections 2024-03-12T03:09:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:14.432+0000 7f42243fc700 1 -- 172.21.15.94:0/2543352548 wait complete. 2024-03-12T03:09:15.213 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-12T03:09:15.213 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-12T03:09:15.657 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:15 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2543352548' entity='client.admin' 2024-03-12T03:09:15.780 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:09:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.099+0000 7f0c844f5700 1 -- 172.21.15.94:0/2764354270 >> v1:172.21.15.94:6789/0 conn(0x7f0c7c100640 legacy=0x7f0c7c100a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.100+0000 7f0c844f5700 1 -- 172.21.15.94:0/2764354270 shutdown_connections 2024-03-12T03:09:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.100+0000 7f0c844f5700 1 -- 172.21.15.94:0/2764354270 >> 172.21.15.94:0/2764354270 conn(0x7f0c7c0fbbe0 msgr2=0x7f0c7c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.100+0000 7f0c844f5700 1 -- 172.21.15.94:0/2764354270 shutdown_connections 2024-03-12T03:09:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.100+0000 7f0c844f5700 1 -- 172.21.15.94:0/2764354270 wait complete. 2024-03-12T03:09:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.100+0000 7f0c844f5700 1 Processor -- start 2024-03-12T03:09:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.100+0000 7f0c844f5700 1 -- start start 2024-03-12T03:09:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.100+0000 7f0c844f5700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0c7c1968c0 con 0x7f0c7c100640 2024-03-12T03:09:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.100+0000 7f0c82291700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0c7c100640 0x7f0c7c1961b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43836/0 (socket says 172.21.15.94:43836) 2024-03-12T03:09:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.100+0000 7f0c82291700 1 -- 172.21.15.94:0/1240708490 learned_addr learned my addr 172.21.15.94:0/1240708490 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.101+0000 7f0c737fe700 1 -- 172.21.15.94:0/1240708490 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256821073 0 0) 0x7f0c7c1968c0 con 0x7f0c7c100640 2024-03-12T03:09:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.101+0000 7f0c737fe700 1 -- 172.21.15.94:0/1240708490 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c68005530 con 0x7f0c7c100640 2024-03-12T03:09:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.101+0000 7f0c737fe700 1 -- 172.21.15.94:0/1240708490 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 652724344 0 0) 0x7f0c68005530 con 0x7f0c7c100640 2024-03-12T03:09:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.101+0000 7f0c737fe700 1 -- 172.21.15.94:0/1240708490 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c7c1968c0 con 0x7f0c7c100640 2024-03-12T03:09:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.101+0000 7f0c737fe700 1 -- 172.21.15.94:0/1240708490 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f0c78001480 con 0x7f0c7c100640 2024-03-12T03:09:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.101+0000 7f0c737fe700 1 -- 172.21.15.94:0/1240708490 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2885331070 0 0) 0x7f0c7c1968c0 con 0x7f0c7c100640 2024-03-12T03:09:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.101+0000 7f0c737fe700 1 -- 172.21.15.94:0/1240708490 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7c196a90 con 0x7f0c7c100640 2024-03-12T03:09:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.101+0000 7f0c844f5700 1 -- 172.21.15.94:0/1240708490 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7c196d80 con 0x7f0c7c100640 2024-03-12T03:09:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.101+0000 7f0c844f5700 1 -- 172.21.15.94:0/1240708490 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c7c19a8d0 con 0x7f0c7c100640 2024-03-12T03:09:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.101+0000 7f0c737fe700 1 -- 172.21.15.94:0/1240708490 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f0c780036e0 con 0x7f0c7c100640 2024-03-12T03:09:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.101+0000 7f0c737fe700 1 -- 172.21.15.94:0/1240708490 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f0c78004d20 con 0x7f0c7c100640 2024-03-12T03:09:18.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.102+0000 7f0c737fe700 1 -- 172.21.15.94:0/1240708490 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7f0c78004ee0 con 0x7f0c7c100640 2024-03-12T03:09:18.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.102+0000 7f0c844f5700 1 -- 172.21.15.94:0/1240708490 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c60005150 con 0x7f0c7c100640 2024-03-12T03:09:18.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.102+0000 7f0c737fe700 1 -- 172.21.15.94:0/1240708490 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 2889753012 0 0) 0x7f0c78047790 con 0x7f0c7c100640 2024-03-12T03:09:18.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.107+0000 7f0c737fe700 1 -- 172.21.15.94:0/1240708490 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f0c78017480 con 0x7f0c7c100640 2024-03-12T03:09:18.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.306+0000 7f0c844f5700 1 -- 172.21.15.94:0/1240708490 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c60000bc0 con 0x7f0c6803a2f0 2024-03-12T03:09:18.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.309+0000 7f0c737fe700 1 -- 172.21.15.94:0/1240708490 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7f0c60000bc0 con 0x7f0c6803a2f0 2024-03-12T03:09:18.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.311+0000 7f0c844f5700 1 -- 172.21.15.94:0/1240708490 >> v1:172.21.15.94:6800/1299282424 conn(0x7f0c6803a2f0 legacy=0x7f0c6803c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:18.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.311+0000 7f0c844f5700 1 -- 172.21.15.94:0/1240708490 >> v1:172.21.15.94:6789/0 conn(0x7f0c7c100640 legacy=0x7f0c7c1961b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:18.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.311+0000 7f0c844f5700 1 -- 172.21.15.94:0/1240708490 shutdown_connections 2024-03-12T03:09:18.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.311+0000 7f0c844f5700 1 -- 172.21.15.94:0/1240708490 >> 172.21.15.94:0/1240708490 conn(0x7f0c7c0fbbe0 msgr2=0x7f0c7c102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:18.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.311+0000 7f0c844f5700 1 -- 172.21.15.94:0/1240708490 shutdown_connections 2024-03-12T03:09:18.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:18.311+0000 7f0c844f5700 1 -- 172.21.15.94:0/1240708490 wait complete. 2024-03-12T03:09:18.696 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:18 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:18.696 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:18 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:18.696 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:18 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi094", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:09:18.696 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:18 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:18.696 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:18 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:09:18.696 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:18 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:19.094 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi151 2024-03-12T03:09:19.094 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-12T03:09:19.094 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/etc/ceph/ceph.conf 2024-03-12T03:09:19.113 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-12T03:09:19.113 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:09:19.170 INFO:tasks.cephadm:Adding host smithi151 to orchestrator... 2024-03-12T03:09:19.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch host add smithi151 2024-03-12T03:09:19.408 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:09:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:19 smithi094 ceph-mon[29291]: from='client.14176 v1:172.21.15.94:0/1240708490' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:09:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:19 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:19 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:09:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:19 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:19 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:09:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:19 smithi094 ceph-mon[29291]: Updating smithi094:/etc/ceph/ceph.conf 2024-03-12T03:09:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:19 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:19 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:19 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:20.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:20 smithi094 ceph-mon[29291]: Updating smithi094:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:09:20.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:20 smithi094 ceph-mon[29291]: Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:09:20.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:20 smithi094 ceph-mon[29291]: Updating smithi094:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:09:21.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.085+0000 7fba94895700 1 -- 172.21.15.94:0/642356053 >> v1:172.21.15.94:6789/0 conn(0x7fba90106690 legacy=0x7fba90106a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:21.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.085+0000 7fba94895700 1 -- 172.21.15.94:0/642356053 shutdown_connections 2024-03-12T03:09:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.085+0000 7fba94895700 1 -- 172.21.15.94:0/642356053 >> 172.21.15.94:0/642356053 conn(0x7fba900fc110 msgr2=0x7fba900fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.085+0000 7fba94895700 1 -- 172.21.15.94:0/642356053 shutdown_connections 2024-03-12T03:09:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.086+0000 7fba94895700 1 -- 172.21.15.94:0/642356053 wait complete. 2024-03-12T03:09:21.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.086+0000 7fba94895700 1 Processor -- start 2024-03-12T03:09:21.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.086+0000 7fba94895700 1 -- start start 2024-03-12T03:09:21.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.086+0000 7fba94895700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba90195c90 con 0x7fba90106690 2024-03-12T03:09:21.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.087+0000 7fba8e59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fba90106690 0x7fba90195580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43850/0 (socket says 172.21.15.94:43850) 2024-03-12T03:09:21.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.087+0000 7fba8e59c700 1 -- 172.21.15.94:0/1793023473 learned_addr learned my addr 172.21.15.94:0/1793023473 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:21.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.087+0000 7fba7f7fe700 1 -- 172.21.15.94:0/1793023473 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149493447 0 0) 0x7fba90195c90 con 0x7fba90106690 2024-03-12T03:09:21.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.087+0000 7fba7f7fe700 1 -- 172.21.15.94:0/1793023473 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba78005530 con 0x7fba90106690 2024-03-12T03:09:21.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.087+0000 7fba7f7fe700 1 -- 172.21.15.94:0/1793023473 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3748500093 0 0) 0x7fba78005530 con 0x7fba90106690 2024-03-12T03:09:21.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.087+0000 7fba7f7fe700 1 -- 172.21.15.94:0/1793023473 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba90195c90 con 0x7fba90106690 2024-03-12T03:09:21.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.087+0000 7fba7f7fe700 1 -- 172.21.15.94:0/1793023473 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fba800047e0 con 0x7fba90106690 2024-03-12T03:09:21.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.087+0000 7fba7f7fe700 1 -- 172.21.15.94:0/1793023473 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3282710109 0 0) 0x7fba90195c90 con 0x7fba90106690 2024-03-12T03:09:21.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.087+0000 7fba7f7fe700 1 -- 172.21.15.94:0/1793023473 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba90195e60 con 0x7fba90106690 2024-03-12T03:09:21.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.087+0000 7fba94895700 1 -- 172.21.15.94:0/1793023473 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba901960f0 con 0x7fba90106690 2024-03-12T03:09:21.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.087+0000 7fba94895700 1 -- 172.21.15.94:0/1793023473 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba90069ec0 con 0x7fba90106690 2024-03-12T03:09:21.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.087+0000 7fba7f7fe700 1 -- 172.21.15.94:0/1793023473 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fba80001a80 con 0x7fba90106690 2024-03-12T03:09:21.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.087+0000 7fba7f7fe700 1 -- 172.21.15.94:0/1793023473 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fba80004d30 con 0x7fba90106690 2024-03-12T03:09:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.088+0000 7fba94895700 1 -- 172.21.15.94:0/1793023473 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba70005120 con 0x7fba90106690 2024-03-12T03:09:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.088+0000 7fba7f7fe700 1 -- 172.21.15.94:0/1793023473 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7fba80004fb0 con 0x7fba90106690 2024-03-12T03:09:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.088+0000 7fba7f7fe700 1 -- 172.21.15.94:0/1793023473 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 2889753012 0 0) 0x7fba80047690 con 0x7fba90106690 2024-03-12T03:09:21.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.093+0000 7fba7f7fe700 1 -- 172.21.15.94:0/1793023473 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fba90107210 con 0x7fba90106690 2024-03-12T03:09:21.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:21.292+0000 7fba94895700 1 -- 172.21.15.94:0/1793023473 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi151", "target": ["mon-mgr", ""]}) v1 -- 0x7fba70000b90 con 0x7fba7803c4f0 2024-03-12T03:09:21.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:21 smithi094 ceph-mon[29291]: from='client.14178 v1:172.21.15.94:0/1793023473' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi151", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:09:22.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:22 smithi094 ceph-mon[29291]: Deploying cephadm binary to smithi151 2024-03-12T03:09:22.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:22 smithi094 ceph-mon[29291]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:25.619 INFO:teuthology.orchestra.run.smithi094.stdout:Added host 'smithi151' with addr '172.21.15.151' 2024-03-12T03:09:25.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:25.616+0000 7fba7f7fe700 1 -- 172.21.15.94:0/1793023473 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (unknown 0 0 853843814) 0x7fba70000b90 con 0x7fba7803c4f0 2024-03-12T03:09:25.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:25.618+0000 7fba94895700 1 -- 172.21.15.94:0/1793023473 >> v1:172.21.15.94:6800/1299282424 conn(0x7fba7803c4f0 legacy=0x7fba7803e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:25.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:25.618+0000 7fba94895700 1 -- 172.21.15.94:0/1793023473 >> v1:172.21.15.94:6789/0 conn(0x7fba90106690 legacy=0x7fba90195580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:25.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:25.618+0000 7fba94895700 1 -- 172.21.15.94:0/1793023473 shutdown_connections 2024-03-12T03:09:25.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:25.618+0000 7fba94895700 1 -- 172.21.15.94:0/1793023473 >> 172.21.15.94:0/1793023473 conn(0x7fba900fc110 msgr2=0x7fba90107780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:25.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:25.619+0000 7fba94895700 1 -- 172.21.15.94:0/1793023473 shutdown_connections 2024-03-12T03:09:25.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:25.619+0000 7fba94895700 1 -- 172.21.15.94:0/1793023473 wait complete. 2024-03-12T03:09:25.633 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:25 smithi094 ceph-mon[29291]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:26.263 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-12T03:09:26.501 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:09:26.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:26.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:26 smithi094 ceph-mon[29291]: Added host smithi151 2024-03-12T03:09:26.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:09:26.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:26.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:26 smithi094 ceph-mon[29291]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:27.912 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:28.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.141+0000 7f6fa32b9700 1 -- 172.21.15.94:0/1967946065 >> v1:172.21.15.94:6789/0 conn(0x7f6f9c1076c0 legacy=0x7f6f9c107aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:28.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.141+0000 7f6fa32b9700 1 -- 172.21.15.94:0/1967946065 shutdown_connections 2024-03-12T03:09:28.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.141+0000 7f6fa32b9700 1 -- 172.21.15.94:0/1967946065 >> 172.21.15.94:0/1967946065 conn(0x7f6f9c0fd1f0 msgr2=0x7f6f9c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:28.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.141+0000 7f6fa32b9700 1 -- 172.21.15.94:0/1967946065 shutdown_connections 2024-03-12T03:09:28.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.141+0000 7f6fa32b9700 1 -- 172.21.15.94:0/1967946065 wait complete. 2024-03-12T03:09:28.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.142+0000 7f6fa32b9700 1 Processor -- start 2024-03-12T03:09:28.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.142+0000 7f6fa32b9700 1 -- start start 2024-03-12T03:09:28.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.142+0000 7f6fa32b9700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6f9c196d90 con 0x7f6f9c1076c0 2024-03-12T03:09:28.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.142+0000 7f6fa1055700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6f9c1076c0 0x7f6f9c196680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36152/0 (socket says 172.21.15.94:36152) 2024-03-12T03:09:28.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.142+0000 7f6fa1055700 1 -- 172.21.15.94:0/370855332 learned_addr learned my addr 172.21.15.94:0/370855332 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:28.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.142+0000 7f6f927fc700 1 -- 172.21.15.94:0/370855332 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847009841 0 0) 0x7f6f9c196d90 con 0x7f6f9c1076c0 2024-03-12T03:09:28.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.143+0000 7f6f927fc700 1 -- 172.21.15.94:0/370855332 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f88005530 con 0x7f6f9c1076c0 2024-03-12T03:09:28.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.143+0000 7f6f927fc700 1 -- 172.21.15.94:0/370855332 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3440643631 0 0) 0x7f6f88005530 con 0x7f6f9c1076c0 2024-03-12T03:09:28.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.143+0000 7f6f927fc700 1 -- 172.21.15.94:0/370855332 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f9c196d90 con 0x7f6f9c1076c0 2024-03-12T03:09:28.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.143+0000 7f6f927fc700 1 -- 172.21.15.94:0/370855332 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f6f980047e0 con 0x7f6f9c1076c0 2024-03-12T03:09:28.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.143+0000 7f6f927fc700 1 -- 172.21.15.94:0/370855332 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1055098785 0 0) 0x7f6f9c196d90 con 0x7f6f9c1076c0 2024-03-12T03:09:28.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.143+0000 7f6f927fc700 1 -- 172.21.15.94:0/370855332 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c196f60 con 0x7f6f9c1076c0 2024-03-12T03:09:28.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.143+0000 7f6fa32b9700 1 -- 172.21.15.94:0/370855332 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9c197250 con 0x7f6f9c1076c0 2024-03-12T03:09:28.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.143+0000 7f6fa32b9700 1 -- 172.21.15.94:0/370855332 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9c19f410 con 0x7f6f9c1076c0 2024-03-12T03:09:28.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.143+0000 7f6f927fc700 1 -- 172.21.15.94:0/370855332 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f6f98001a80 con 0x7f6f9c1076c0 2024-03-12T03:09:28.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.143+0000 7f6f927fc700 1 -- 172.21.15.94:0/370855332 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f6f98004d40 con 0x7f6f9c1076c0 2024-03-12T03:09:28.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.144+0000 7f6f927fc700 1 -- 172.21.15.94:0/370855332 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7f6f98004f00 con 0x7f6f9c1076c0 2024-03-12T03:09:28.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.144+0000 7f6fa32b9700 1 -- 172.21.15.94:0/370855332 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f80005120 con 0x7f6f9c1076c0 2024-03-12T03:09:28.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.144+0000 7f6f927fc700 1 -- 172.21.15.94:0/370855332 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 2889753012 0 0) 0x7f6f98047590 con 0x7f6f9c1076c0 2024-03-12T03:09:28.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.149+0000 7f6f927fc700 1 -- 172.21.15.94:0/370855332 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f6f9c108240 con 0x7f6f9c1076c0 2024-03-12T03:09:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.348+0000 7f6fa32b9700 1 -- 172.21.15.94:0/370855332 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6f80000b90 con 0x7f6f8803a3e0 2024-03-12T03:09:28.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.349+0000 7f6f927fc700 1 -- 172.21.15.94:0/370855332 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (unknown 0 0 3262600364) 0x7f6f80000b90 con 0x7f6f8803a3e0 2024-03-12T03:09:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:09:28.352 INFO:teuthology.orchestra.run.smithi094.stdout:[{"addr": "172.21.15.94", "hostname": "smithi094", "labels": [], "status": ""}, {"addr": "172.21.15.151", "hostname": "smithi151", "labels": [], "status": ""}] 2024-03-12T03:09:28.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.351+0000 7f6fa32b9700 1 -- 172.21.15.94:0/370855332 >> v1:172.21.15.94:6800/1299282424 conn(0x7f6f8803a3e0 legacy=0x7f6f8803c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:28.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.351+0000 7f6fa32b9700 1 -- 172.21.15.94:0/370855332 >> v1:172.21.15.94:6789/0 conn(0x7f6f9c1076c0 legacy=0x7f6f9c196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:28.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.351+0000 7f6fa32b9700 1 -- 172.21.15.94:0/370855332 shutdown_connections 2024-03-12T03:09:28.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.351+0000 7f6fa32b9700 1 -- 172.21.15.94:0/370855332 >> 172.21.15.94:0/370855332 conn(0x7f6f9c0fd1f0 msgr2=0x7f6f9c0ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:28.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.351+0000 7f6fa32b9700 1 -- 172.21.15.94:0/370855332 shutdown_connections 2024-03-12T03:09:28.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:28.351+0000 7f6fa32b9700 1 -- 172.21.15.94:0/370855332 wait complete. 2024-03-12T03:09:28.933 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-12T03:09:28.934 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-12T03:09:29.168 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:09:29.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:28 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:29.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:28 smithi094 ceph-mon[29291]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:29.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:28 smithi094 ceph-mon[29291]: from='client.14180 v1:172.21.15.94:0/370855332' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:09:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.685+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/2363993437 >> v1:172.21.15.94:6789/0 conn(0x7ff8d8106620 legacy=0x7ff8d8106a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.685+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/2363993437 shutdown_connections 2024-03-12T03:09:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.685+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/2363993437 >> 172.21.15.94:0/2363993437 conn(0x7ff8d80fc130 msgr2=0x7ff8d80fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.685+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/2363993437 shutdown_connections 2024-03-12T03:09:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.685+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/2363993437 wait complete. 2024-03-12T03:09:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.686+0000 7ff8dcfcc700 1 Processor -- start 2024-03-12T03:09:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.686+0000 7ff8dcfcc700 1 -- start start 2024-03-12T03:09:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.686+0000 7ff8dcfcc700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff8d8196e40 con 0x7ff8d8106620 2024-03-12T03:09:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.687+0000 7ff8d659c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff8d8106620 0x7ff8d8196730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36164/0 (socket says 172.21.15.94:36164) 2024-03-12T03:09:30.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.687+0000 7ff8d659c700 1 -- 172.21.15.94:0/209802826 learned_addr learned my addr 172.21.15.94:0/209802826 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:30.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.687+0000 7ff8c77fe700 1 -- 172.21.15.94:0/209802826 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808941151 0 0) 0x7ff8d8196e40 con 0x7ff8d8106620 2024-03-12T03:09:30.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.687+0000 7ff8c77fe700 1 -- 172.21.15.94:0/209802826 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8c0005530 con 0x7ff8d8106620 2024-03-12T03:09:30.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.687+0000 7ff8c77fe700 1 -- 172.21.15.94:0/209802826 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3984428043 0 0) 0x7ff8c0005530 con 0x7ff8d8106620 2024-03-12T03:09:30.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.687+0000 7ff8c77fe700 1 -- 172.21.15.94:0/209802826 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8d8196e40 con 0x7ff8d8106620 2024-03-12T03:09:30.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.687+0000 7ff8c77fe700 1 -- 172.21.15.94:0/209802826 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7ff8c80047e0 con 0x7ff8d8106620 2024-03-12T03:09:30.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.687+0000 7ff8c77fe700 1 -- 172.21.15.94:0/209802826 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 116049471 0 0) 0x7ff8d8196e40 con 0x7ff8d8106620 2024-03-12T03:09:30.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.687+0000 7ff8c77fe700 1 -- 172.21.15.94:0/209802826 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d8197010 con 0x7ff8d8106620 2024-03-12T03:09:30.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.687+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/209802826 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d8197300 con 0x7ff8d8106620 2024-03-12T03:09:30.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.687+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/209802826 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d819f4c0 con 0x7ff8d8106620 2024-03-12T03:09:30.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.687+0000 7ff8c77fe700 1 -- 172.21.15.94:0/209802826 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7ff8c8001a80 con 0x7ff8d8106620 2024-03-12T03:09:30.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.687+0000 7ff8c77fe700 1 -- 172.21.15.94:0/209802826 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7ff8c8004dc0 con 0x7ff8d8106620 2024-03-12T03:09:30.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.688+0000 7ff8c77fe700 1 -- 172.21.15.94:0/209802826 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7ff8c8005000 con 0x7ff8d8106620 2024-03-12T03:09:30.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.688+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/209802826 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b8005120 con 0x7ff8d8106620 2024-03-12T03:09:30.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.688+0000 7ff8c77fe700 1 -- 172.21.15.94:0/209802826 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 2889753012 0 0) 0x7ff8c8047690 con 0x7ff8d8106620 2024-03-12T03:09:30.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.693+0000 7ff8c77fe700 1 -- 172.21.15.94:0/209802826 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7ff8d81071a0 con 0x7ff8d8106620 2024-03-12T03:09:30.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:30.894+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/209802826 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff8b8005410 con 0x7ff8d8106620 2024-03-12T03:09:31.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:31 smithi094 ceph-mon[29291]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:31 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:31 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:31 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:31 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:31 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:09:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:31 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:31 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:09:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/209802826' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T03:09:31.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:31.791+0000 7ff8c77fe700 1 -- 172.21.15.94:0/209802826 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (unknown 3126668360 0 0) 0x7ff8c8017550 con 0x7ff8d8106620 2024-03-12T03:09:31.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:31.793+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/209802826 >> v1:172.21.15.94:6800/1299282424 conn(0x7ff8c003a2f0 legacy=0x7ff8c003c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:31.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:31.793+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/209802826 >> v1:172.21.15.94:6789/0 conn(0x7ff8d8106620 legacy=0x7ff8d8196730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:31.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:31.793+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/209802826 shutdown_connections 2024-03-12T03:09:31.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:31.793+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/209802826 >> 172.21.15.94:0/209802826 conn(0x7ff8d80fc130 msgr2=0x7ff8d8069d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:31.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:31.793+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/209802826 shutdown_connections 2024-03-12T03:09:31.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:31.793+0000 7ff8dcfcc700 1 -- 172.21.15.94:0/209802826 wait complete. 2024-03-12T03:09:31.797 INFO:teuthology.orchestra.run.smithi094.stderr:adjusted tunables profile to default 2024-03-12T03:09:32.399 INFO:tasks.cephadm:Adding mon.a on smithi094 2024-03-12T03:09:32.399 INFO:tasks.cephadm:Adding mon.c on smithi094 2024-03-12T03:09:32.399 INFO:tasks.cephadm:Adding mon.b on smithi151 2024-03-12T03:09:32.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi094:[v1:172.21.15.94:6789]=a;smithi094:[v1:172.21.15.94:6790]=c;smithi151:[v1:172.21.15.151:6789]=b' 2024-03-12T03:09:32.626 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:09:32.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:32 smithi094 ceph-mon[29291]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-12T03:09:32.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:32 smithi094 ceph-mon[29291]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:09:32.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:32 smithi094 ceph-mon[29291]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:09:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:32 smithi094 ceph-mon[29291]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:09:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:32 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:32 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:32 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/209802826' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T03:09:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:32 smithi094 ceph-mon[29291]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:09:33.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:33 smithi094 ceph-mon[29291]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:34.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.068+0000 7fbce209f700 1 -- 172.21.15.151:0/4014249816 >> v1:172.21.15.94:6789/0 conn(0x7fbcdc106680 legacy=0x7fbcdc106a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:34.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.069+0000 7fbce209f700 1 -- 172.21.15.151:0/4014249816 shutdown_connections 2024-03-12T03:09:34.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.069+0000 7fbce209f700 1 -- 172.21.15.151:0/4014249816 >> 172.21.15.151:0/4014249816 conn(0x7fbcdc0fc100 msgr2=0x7fbcdc0fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:34.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.069+0000 7fbce209f700 1 -- 172.21.15.151:0/4014249816 shutdown_connections 2024-03-12T03:09:34.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.069+0000 7fbce209f700 1 -- 172.21.15.151:0/4014249816 wait complete. 2024-03-12T03:09:34.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.069+0000 7fbce209f700 1 Processor -- start 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.069+0000 7fbce209f700 1 -- start start 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.069+0000 7fbce209f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbcdc196d90 con 0x7fbcdc106680 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.070+0000 7fbcdb7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbcdc106680 0x7fbcdc196680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.151:53252/0 (socket says 172.21.15.151:53252) 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.070+0000 7fbcdb7fe700 1 -- 172.21.15.151:0/694102970 learned_addr learned my addr 172.21.15.151:0/694102970 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.070+0000 7fbcd8ff9700 1 -- 172.21.15.151:0/694102970 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238836233 0 0) 0x7fbcdc196d90 con 0x7fbcdc106680 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.070+0000 7fbcd8ff9700 1 -- 172.21.15.151:0/694102970 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcc4005530 con 0x7fbcdc106680 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.070+0000 7fbcd8ff9700 1 -- 172.21.15.151:0/694102970 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3774027715 0 0) 0x7fbcc4005530 con 0x7fbcdc106680 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.070+0000 7fbcd8ff9700 1 -- 172.21.15.151:0/694102970 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcdc196d90 con 0x7fbcdc106680 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.070+0000 7fbcd8ff9700 1 -- 172.21.15.151:0/694102970 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fbcd00047e0 con 0x7fbcdc106680 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.070+0000 7fbcd8ff9700 1 -- 172.21.15.151:0/694102970 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 441137196 0 0) 0x7fbcdc196d90 con 0x7fbcdc106680 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.070+0000 7fbcd8ff9700 1 -- 172.21.15.151:0/694102970 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc196f60 con 0x7fbcdc106680 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.070+0000 7fbce209f700 1 -- 172.21.15.151:0/694102970 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcdc197250 con 0x7fbcdc106680 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.070+0000 7fbce209f700 1 -- 172.21.15.151:0/694102970 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcdc19f490 con 0x7fbcdc106680 2024-03-12T03:09:34.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.070+0000 7fbcd8ff9700 1 -- 172.21.15.151:0/694102970 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fbcd0001a80 con 0x7fbcdc106680 2024-03-12T03:09:34.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.070+0000 7fbcd8ff9700 1 -- 172.21.15.151:0/694102970 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7fbcd0004d40 con 0x7fbcdc106680 2024-03-12T03:09:34.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.071+0000 7fbcd8ff9700 1 -- 172.21.15.151:0/694102970 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7fbcd0004f00 con 0x7fbcdc106680 2024-03-12T03:09:34.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.071+0000 7fbce209f700 1 -- 172.21.15.151:0/694102970 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc8005120 con 0x7fbcdc106680 2024-03-12T03:09:34.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.071+0000 7fbcd8ff9700 1 -- 172.21.15.151:0/694102970 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 3179478896 0 0) 0x7fbcd0047880 con 0x7fbcdc106680 2024-03-12T03:09:34.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.077+0000 7fbcd8ff9700 1 -- 172.21.15.151:0/694102970 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fbcdc107200 con 0x7fbcdc106680 2024-03-12T03:09:34.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.282+0000 7fbce209f700 1 -- 172.21.15.151:0/694102970 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi094:[v1:172.21.15.94:6789]=a;smithi094:[v1:172.21.15.94:6790]=c;smithi151:[v1:172.21.15.151:6789]=b", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcc8000c60 con 0x7fbcc403a330 2024-03-12T03:09:34.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.288+0000 7fbcd8ff9700 1 -- 172.21.15.151:0/694102970 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 3265049985) 0x7fbcc8000c60 con 0x7fbcc403a330 2024-03-12T03:09:34.291 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled mon update... 2024-03-12T03:09:34.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.290+0000 7fbce209f700 1 -- 172.21.15.151:0/694102970 >> v1:172.21.15.94:6800/1299282424 conn(0x7fbcc403a330 legacy=0x7fbcc403c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:34.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.290+0000 7fbce209f700 1 -- 172.21.15.151:0/694102970 >> v1:172.21.15.94:6789/0 conn(0x7fbcdc106680 legacy=0x7fbcdc196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:34.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.290+0000 7fbce209f700 1 -- 172.21.15.151:0/694102970 shutdown_connections 2024-03-12T03:09:34.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.290+0000 7fbce209f700 1 -- 172.21.15.151:0/694102970 >> 172.21.15.151:0/694102970 conn(0x7fbcdc0fc100 msgr2=0x7fbcdc069d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:34.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.290+0000 7fbce209f700 1 -- 172.21.15.151:0/694102970 shutdown_connections 2024-03-12T03:09:34.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:34.290+0000 7fbce209f700 1 -- 172.21.15.151:0/694102970 wait complete. 2024-03-12T03:09:34.869 DEBUG:teuthology.orchestra.run.smithi094:mon.c> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.c.service 2024-03-12T03:09:34.872 DEBUG:teuthology.orchestra.run.smithi151:mon.b> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.b.service 2024-03-12T03:09:34.874 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T03:09:34.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:09:35.663 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:09:35.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:35 smithi094 ceph-mon[29291]: from='client.14184 v1:172.21.15.151:0/694102970' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi094:[v1:172.21.15.94:6789]=a;smithi094:[v1:172.21.15.94:6790]=c;smithi151:[v1:172.21.15.151:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:09:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:35 smithi094 ceph-mon[29291]: Saving service mon spec with placement smithi094:[v1:172.21.15.94:6789]=a;smithi094:[v1:172.21.15.94:6790]=c;smithi151:[v1:172.21.15.151:6789]=b;count:3 2024-03-12T03:09:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:35 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:35 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:09:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:35 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:35 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:09:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:35 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:35 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:09:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:35 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:35 smithi094 ceph-mon[29291]: Deploying daemon mon.b on smithi151 2024-03-12T03:09:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:35 smithi094 ceph-mon[29291]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:37.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:37 smithi094 ceph-mon[29291]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:38.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.710+0000 7f4df6ed3700 1 -- 172.21.15.151:0/786692124 >> v1:172.21.15.94:6789/0 conn(0x7f4df0106650 legacy=0x7f4df0106a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:38.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.710+0000 7f4df6ed3700 1 -- 172.21.15.151:0/786692124 shutdown_connections 2024-03-12T03:09:38.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.710+0000 7f4df6ed3700 1 -- 172.21.15.151:0/786692124 >> 172.21.15.151:0/786692124 conn(0x7f4df00fc0f0 msgr2=0x7f4df00fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:38.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.710+0000 7f4df6ed3700 1 -- 172.21.15.151:0/786692124 shutdown_connections 2024-03-12T03:09:38.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.710+0000 7f4df6ed3700 1 -- 172.21.15.151:0/786692124 wait complete. 2024-03-12T03:09:38.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.711+0000 7f4df6ed3700 1 Processor -- start 2024-03-12T03:09:38.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.711+0000 7f4df6ed3700 1 -- start start 2024-03-12T03:09:38.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.711+0000 7f4df6ed3700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4df0197bd0 con 0x7f4df0106650 2024-03-12T03:09:38.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.711+0000 7f4df4c6f700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4df0106650 0x7f4df01974c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.151:60538/0 (socket says 172.21.15.151:60538) 2024-03-12T03:09:38.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.711+0000 7f4df4c6f700 1 -- 172.21.15.151:0/145405499 learned_addr learned my addr 172.21.15.151:0/145405499 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:09:38.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.712+0000 7f4dedffb700 1 -- 172.21.15.151:0/145405499 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715765358 0 0) 0x7f4df0197bd0 con 0x7f4df0106650 2024-03-12T03:09:38.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.712+0000 7f4dedffb700 1 -- 172.21.15.151:0/145405499 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4de0005530 con 0x7f4df0106650 2024-03-12T03:09:38.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.712+0000 7f4dedffb700 1 -- 172.21.15.151:0/145405499 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1893089649 0 0) 0x7f4de0005530 con 0x7f4df0106650 2024-03-12T03:09:38.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.712+0000 7f4dedffb700 1 -- 172.21.15.151:0/145405499 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4df0197bd0 con 0x7f4df0106650 2024-03-12T03:09:38.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.712+0000 7f4dedffb700 1 -- 172.21.15.151:0/145405499 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f4de40047e0 con 0x7f4df0106650 2024-03-12T03:09:38.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.712+0000 7f4dedffb700 1 -- 172.21.15.151:0/145405499 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1251805064 0 0) 0x7f4df0197bd0 con 0x7f4df0106650 2024-03-12T03:09:38.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.712+0000 7f4dedffb700 1 -- 172.21.15.151:0/145405499 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0197da0 con 0x7f4df0106650 2024-03-12T03:09:38.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.712+0000 7f4df6ed3700 1 -- 172.21.15.151:0/145405499 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df0198090 con 0x7f4df0106650 2024-03-12T03:09:38.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.712+0000 7f4dedffb700 1 -- 172.21.15.151:0/145405499 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f4de4003b10 con 0x7f4df0106650 2024-03-12T03:09:38.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.712+0000 7f4dedffb700 1 -- 172.21.15.151:0/145405499 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3084945678 0 0) 0x7f4de4005000 con 0x7f4df0106650 2024-03-12T03:09:38.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.712+0000 7f4df6ed3700 1 -- 172.21.15.151:0/145405499 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df019f700 con 0x7f4df0106650 2024-03-12T03:09:38.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.713+0000 7f4dedffb700 1 -- 172.21.15.151:0/145405499 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7f4de40051e0 con 0x7f4df0106650 2024-03-12T03:09:38.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.713+0000 7f4dedffb700 1 -- 172.21.15.151:0/145405499 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 3179478896 0 0) 0x7f4de40477c0 con 0x7f4df0106650 2024-03-12T03:09:38.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.714+0000 7f4df6ed3700 1 -- 172.21.15.151:0/145405499 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df0101d00 con 0x7f4df0106650 2024-03-12T03:09:38.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:38.719+0000 7f4dedffb700 1 -- 172.21.15.151:0/145405499 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f4df01071d0 con 0x7f4df0106650 2024-03-12T03:09:39.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:39.031+0000 7f4df6ed3700 1 -- 172.21.15.151:0/145405499 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4df006a2c0 con 0x7f4df0106650 2024-03-12T03:09:39.035 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:38 smithi151 podman[32888]: 2024-03-12 03:09:38.985705159 +0000 UTC m=+0.119850003 container create 5f664b44d1d7bff77c00d729ff0229587d238140b4f14627a95255c59b9657c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-12T03:09:39.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:39.032+0000 7f4dedffb700 1 -- 172.21.15.151:0/145405499 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (unknown 2237029548 0 456308052) 0x7f4de4017470 con 0x7f4df0106650 2024-03-12T03:09:39.035 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:09:39.035 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":1,"fsid":"964142e8-e01d-11ee-95c8-87774f69a715","modified":"2024-03-12T03:08:04.720419Z","created":"2024-03-12T03:08:04.720419Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:09:39.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:39.034+0000 7f4df6ed3700 1 -- 172.21.15.151:0/145405499 >> v1:172.21.15.94:6800/1299282424 conn(0x7f4de003a330 legacy=0x7f4de003c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:39.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:39.034+0000 7f4df6ed3700 1 -- 172.21.15.151:0/145405499 >> v1:172.21.15.94:6789/0 conn(0x7f4df0106650 legacy=0x7f4df01974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:39.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:39.034+0000 7f4df6ed3700 1 -- 172.21.15.151:0/145405499 shutdown_connections 2024-03-12T03:09:39.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:39.034+0000 7f4df6ed3700 1 -- 172.21.15.151:0/145405499 >> 172.21.15.151:0/145405499 conn(0x7f4df00fc0f0 msgr2=0x7f4df01077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:39.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:39.034+0000 7f4df6ed3700 1 -- 172.21.15.151:0/145405499 shutdown_connections 2024-03-12T03:09:39.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:39.034+0000 7f4df6ed3700 1 -- 172.21.15.151:0/145405499 wait complete. 2024-03-12T03:09:39.037 INFO:teuthology.orchestra.run.smithi151.stderr:dumped monmap epoch 1 2024-03-12T03:09:39.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:39 smithi094 ceph-mon[29291]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:39.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/145405499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:09:39.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 podman[32888]: 2024-03-12 03:09:39.699635719 +0000 UTC m=+0.833780567 container init 5f664b44d1d7bff77c00d729ff0229587d238140b4f14627a95255c59b9657c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-12T03:09:39.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 podman[32888]: 2024-03-12 03:09:39.702698734 +0000 UTC m=+0.836843573 container start 5f664b44d1d7bff77c00d729ff0229587d238140b4f14627a95255c59b9657c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-12T03:09:39.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:09:39.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 2 2024-03-12T03:09:39.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: pidfile_write: ignore empty --pid-file 2024-03-12T03:09:39.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: load: jerasure load: lrc 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: RocksDB version: 7.9.2 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Git sha 0 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: DB SUMMARY 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: DB Session ID: PEYNWZQGUUHXWKH7JI0B 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 476 ; 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:09:39.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T03:09:39.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.env: 0x55f69daa4b20 2024-03-12T03:09:39.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T03:09:39.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.info_log: 0x55f69f751920 2024-03-12T03:09:39.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:09:39.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.statistics: (nil) 2024-03-12T03:09:39.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.use_fsync: 0 2024-03-12T03:09:39.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:09:39.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:09:39.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:09:39.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:09:39.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:09:39.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:09:39.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:09:39.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:09:39.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:09:39.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.db_log_dir: 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.wal_dir: 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.write_buffer_manager: 0x55f69f725400 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:09:39.969 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.unordered_write: 0 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.row_cache: None 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.wal_filter: None 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:09:39.970 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.wal_compression: 0 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:09:39.971 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_open_files: -1 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:09:39.972 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Compression algorithms supported: 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: kZSTD supported: 0 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.merge_operator: 2024-03-12T03:09:39.973 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compaction_filter: None 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f69f751a20) 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout: index_type: 0 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout: data_block_index_type: 0 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout: index_shortening: 1 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout: checksum: 4 2024-03-12T03:09:39.974 INFO:journalctl@ceph.mon.b.smithi151.stdout: no_block_cache: 0 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: block_cache: 0x55f69f757090 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: block_cache_options: 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: capacity : 536870912 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: num_shard_bits : 4 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: strict_capacity_limit : 0 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: block_cache_compressed: (nil) 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: persistent_cache: (nil) 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: block_size: 4096 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: block_size_deviation: 10 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: block_restart_interval: 16 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: index_block_restart_interval: 1 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: metadata_block_size: 4096 2024-03-12T03:09:39.975 INFO:journalctl@ceph.mon.b.smithi151.stdout: partition_filters: 0 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout: use_delta_encoding: 1 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout: filter_policy: bloomfilter 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout: whole_key_filtering: 1 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout: verify_compression: 0 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout: format_version: 5 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout: enable_index_compression: 1 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout: block_align: 0 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout: max_auto_readahead_size: 262144 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout: prepopulate_block_cache: 0 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout: initial_auto_readahead_size: 8192 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compression: NoCompression 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:09:39.976 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.num_levels: 7 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:09:39.977 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:09:39.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:09:39.979 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:09:39.980 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:09:39.981 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.ttl: 2592000 2024-03-12T03:09:39.981 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:09:39.981 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T03:09:39.981 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 60373e87-ba45-472e-a8be-6fbfc915e15d 2024-03-12T03:09:39.982 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710212979732258, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710212979732675, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1587, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710212979, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "60373e87-ba45-472e-a8be-6fbfc915e15d", "db_session_id": "PEYNWZQGUUHXWKH7JI0B", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710212979732756, "job": 1, "event": "recovery_finished"} 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f69f84a000 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: DB pointer 0x55f69f834000 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout: ** DB Stats ** 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:09:39.983 INFO:journalctl@ceph.mon.b.smithi151.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: ** Compaction Stats [default] ** 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: L0 1/0 1.55 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: Sum 1/0 1.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: ** Compaction Stats [default] ** 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T03:09:39.984 INFO:journalctl@ceph.mon.b.smithi151.stdout: 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout: Block cache BinnedLRUCache@0x55f69f757090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout: 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-12T03:09:39.985 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: using public_addrv v1:172.21.15.151:6789/0 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: starting mon.b rank -1 at public addrs v1:172.21.15.151:6789/0 at bind addrs v1:172.21.15.151:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(???) e0 preinit fsid 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(synchronizing).mds e1 new map 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout: e1 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout: legacy client fscid: -1 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout: 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout: No filesystems configured 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-12T03:09:39.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='client.14180 v1:172.21.15.94:0/370855332' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:39.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/209802826' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/209802826' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='client.14184 v1:172.21.15.151:0/694102970' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi094:[v1:172.21.15.94:6789]=a;smithi094:[v1:172.21.15.94:6790]=c;smithi151:[v1:172.21.15.151:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: Saving service mon spec with placement smithi094:[v1:172.21.15.94:6789]=a;smithi094:[v1:172.21.15.94:6790]=c;smithi151:[v1:172.21.15.151:6789]=b;count:3 2024-03-12T03:09:39.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: Deploying daemon mon.b on smithi151 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/145405499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 ceph-mon[32923]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 bash[32888]: 5f664b44d1d7bff77c00d729ff0229587d238140b4f14627a95255c59b9657c2 2024-03-12T03:09:39.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:39 smithi151 systemd[1]: Started Ceph mon.b for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:09:40.609 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T03:09:40.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:09:40.845 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:09:44.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.801+0000 7fac09b88700 1 -- 172.21.15.151:0/3037088543 >> v1:172.21.15.94:6789/0 conn(0x7fac04106740 legacy=0x7fac04106b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:44.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.801+0000 7fac09b88700 1 -- 172.21.15.151:0/3037088543 shutdown_connections 2024-03-12T03:09:44.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.801+0000 7fac09b88700 1 -- 172.21.15.151:0/3037088543 >> 172.21.15.151:0/3037088543 conn(0x7fac040fc250 msgr2=0x7fac040fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:44.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.801+0000 7fac09b88700 1 -- 172.21.15.151:0/3037088543 shutdown_connections 2024-03-12T03:09:44.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.801+0000 7fac09b88700 1 -- 172.21.15.151:0/3037088543 wait complete. 2024-03-12T03:09:44.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.802+0000 7fac09b88700 1 Processor -- start 2024-03-12T03:09:44.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.802+0000 7fac09b88700 1 -- start start 2024-03-12T03:09:44.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.803+0000 7fac09b88700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fac041a4d40 con 0x7fac0419ee70 2024-03-12T03:09:44.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.803+0000 7fac09b88700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fac041a5f20 con 0x7fac04106740 2024-03-12T03:09:44.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.803+0000 7fac037fe700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fac04106740 0x7fac0419e760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:52122/0 (socket says 172.21.15.151:52122) 2024-03-12T03:09:44.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.803+0000 7fac037fe700 1 -- 172.21.15.151:0/4249169791 learned_addr learned my addr 172.21.15.151:0/4249169791 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:09:44.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.803+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978519971 0 0) 0x7fac041a4d40 con 0x7fac0419ee70 2024-03-12T03:09:44.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.803+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabec0035b0 con 0x7fac0419ee70 2024-03-12T03:09:44.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.804+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2005145157 0 0) 0x7fabec0035b0 con 0x7fac0419ee70 2024-03-12T03:09:44.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.804+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac041a4d40 con 0x7fac0419ee70 2024-03-12T03:09:44.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.804+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 239+0+0 (unknown 1839991146 0 0) 0x7fabf80017e0 con 0x7fac0419ee70 2024-03-12T03:09:44.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.804+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2504586155 0 0) 0x7fac041a4d40 con 0x7fac0419ee70 2024-03-12T03:09:44.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.804+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 >> v1:172.21.15.151:6789/0 conn(0x7fac04106740 legacy=0x7fac0419e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:44.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.804+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac041a7100 con 0x7fac0419ee70 2024-03-12T03:09:44.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.804+0000 7fac09b88700 1 -- 172.21.15.151:0/4249169791 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac041a4f70 con 0x7fac0419ee70 2024-03-12T03:09:44.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.804+0000 7fac09b88700 1 -- 172.21.15.151:0/4249169791 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac041a5470 con 0x7fac0419ee70 2024-03-12T03:09:44.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.804+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fabf8003b10 con 0x7fac0419ee70 2024-03-12T03:09:44.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.804+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 239+0+0 (unknown 1839991146 0 0) 0x7fabf80049c0 con 0x7fac0419ee70 2024-03-12T03:09:44.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.805+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7fabf8004b80 con 0x7fac0419ee70 2024-03-12T03:09:44.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.806+0000 7fac09b88700 1 -- 172.21.15.151:0/4249169791 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf0005120 con 0x7fac0419ee70 2024-03-12T03:09:44.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.806+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 3179478896 0 0) 0x7fabf80471d0 con 0x7fac0419ee70 2024-03-12T03:09:44.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:44.812+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fabf8015fa0 con 0x7fac0419ee70 2024-03-12T03:09:45.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:45.088+0000 7fac09b88700 1 -- 172.21.15.151:0/4249169791 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fabf0005410 con 0x7fac0419ee70 2024-03-12T03:09:45.091 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:09:45.091 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: mon.a calling monitor election 2024-03-12T03:09:45.091 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:45.091 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:45.091 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:45.091 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:45.091 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: mon.b calling monitor election 2024-03-12T03:09:45.091 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:45.091 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:45.091 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:45.091 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:45.092 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:45.092 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-12T03:09:45.092 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: monmap e2: 2 mons at {a=v1:172.21.15.94:6789/0,b=v1:172.21.15.151:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:09:45.092 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: fsmap 2024-03-12T03:09:45.092 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:09:45.092 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: mgrmap e13: y(active, since 42s) 2024-03-12T03:09:45.092 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: overall HEALTH_OK 2024-03-12T03:09:45.092 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:45.092 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:45.092 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:45.092 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:09:45.092 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:45.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:45.088+0000 7fac00ff9700 1 -- 172.21.15.151:0/4249169791 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+912 (unknown 3557084514 0 751556733) 0x7fabf80158a0 con 0x7fac0419ee70 2024-03-12T03:09:45.093 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:09:45.093 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":2,"fsid":"964142e8-e01d-11ee-95c8-87774f69a715","modified":"2024-03-12T03:09:39.752465Z","created":"2024-03-12T03:08:04.720419Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-12T03:09:45.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:45.090+0000 7fac09b88700 1 -- 172.21.15.151:0/4249169791 >> v1:172.21.15.94:6800/1299282424 conn(0x7fabec03a420 legacy=0x7fabec03c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:45.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:45.090+0000 7fac09b88700 1 -- 172.21.15.151:0/4249169791 >> v1:172.21.15.94:6789/0 conn(0x7fac0419ee70 legacy=0x7fac041a3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:45.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:45.090+0000 7fac09b88700 1 -- 172.21.15.151:0/4249169791 shutdown_connections 2024-03-12T03:09:45.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:45.090+0000 7fac09b88700 1 -- 172.21.15.151:0/4249169791 >> 172.21.15.151:0/4249169791 conn(0x7fac040fc250 msgr2=0x7fac04069fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:45.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:45.090+0000 7fac09b88700 1 -- 172.21.15.151:0/4249169791 shutdown_connections 2024-03-12T03:09:45.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:45.090+0000 7fac09b88700 1 -- 172.21.15.151:0/4249169791 wait complete. 2024-03-12T03:09:45.094 INFO:teuthology.orchestra.run.smithi151.stderr:dumped monmap epoch 2 2024-03-12T03:09:45.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:09:45.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: mon.a calling monitor election 2024-03-12T03:09:45.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:45.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:45.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:45.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:45.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: mon.b calling monitor election 2024-03-12T03:09:45.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:45.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:45.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:45.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:45.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:45.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-12T03:09:45.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: monmap e2: 2 mons at {a=v1:172.21.15.94:6789/0,b=v1:172.21.15.151:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:09:45.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: fsmap 2024-03-12T03:09:45.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:09:45.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: mgrmap e13: y(active, since 42s) 2024-03-12T03:09:45.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: overall HEALTH_OK 2024-03-12T03:09:45.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:45.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:45.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:45.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:09:45.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:45 smithi151 ceph-mon[32923]: Deploying daemon mon.c on smithi094 2024-03-12T03:09:45.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/4249169791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:09:45.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:45 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:46.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:45 smithi094 ceph-mon[29291]: Deploying daemon mon.c on smithi094 2024-03-12T03:09:46.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/4249169791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:09:46.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:45 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:46.682 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T03:09:46.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:09:46.926 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:09:46.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:46 smithi094 ceph-mon[29291]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:46 smithi151 ceph-mon[32923]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:48.183 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:47 smithi094 systemd[1]: Starting Ceph mon.c for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:09:48.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.277+0000 7eff2a0d4700 1 -- 172.21.15.151:0/1520907544 >> v1:172.21.15.94:6789/0 conn(0x7eff240ffa70 legacy=0x7eff240ffe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:48.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.277+0000 7eff2a0d4700 1 -- 172.21.15.151:0/1520907544 shutdown_connections 2024-03-12T03:09:48.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.277+0000 7eff2a0d4700 1 -- 172.21.15.151:0/1520907544 >> 172.21.15.151:0/1520907544 conn(0x7eff240fb570 msgr2=0x7eff240fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:48.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.277+0000 7eff2a0d4700 1 -- 172.21.15.151:0/1520907544 shutdown_connections 2024-03-12T03:09:48.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.277+0000 7eff2a0d4700 1 -- 172.21.15.151:0/1520907544 wait complete. 2024-03-12T03:09:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.278+0000 7eff2a0d4700 1 Processor -- start 2024-03-12T03:09:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.278+0000 7eff2a0d4700 1 -- start start 2024-03-12T03:09:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.278+0000 7eff2a0d4700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7eff24101e20 con 0x7eff240ffa70 2024-03-12T03:09:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.278+0000 7eff2a0d4700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7eff24101ff0 con 0x7eff24101990 2024-03-12T03:09:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.278+0000 7eff237fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7eff240ffa70 0x7eff24101280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.151:33884/0 (socket says 172.21.15.151:33884) 2024-03-12T03:09:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.278+0000 7eff237fe700 1 -- 172.21.15.151:0/2270687691 learned_addr learned my addr 172.21.15.151:0/2270687691 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:09:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433204308 0 0) 0x7eff24101e20 con 0x7eff240ffa70 2024-03-12T03:09:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff100035b0 con 0x7eff240ffa70 2024-03-12T03:09:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545275331 0 0) 0x7eff24101ff0 con 0x7eff24101990 2024-03-12T03:09:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff24101e20 con 0x7eff24101990 2024-03-12T03:09:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4144733847 0 0) 0x7eff100035b0 con 0x7eff240ffa70 2024-03-12T03:09:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff24101ff0 con 0x7eff240ffa70 2024-03-12T03:09:48.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 239+0+0 (unknown 1839991146 0 0) 0x7eff14004510 con 0x7eff240ffa70 2024-03-12T03:09:48.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4108472165 0 0) 0x7eff24101e20 con 0x7eff24101990 2024-03-12T03:09:48.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff100035b0 con 0x7eff24101990 2024-03-12T03:09:48.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 239+0+0 (unknown 1839991146 0 0) 0x7eff0c0017e0 con 0x7eff24101990 2024-03-12T03:09:48.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1581672615 0 0) 0x7eff24101ff0 con 0x7eff240ffa70 2024-03-12T03:09:48.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 >> v1:172.21.15.151:6789/0 conn(0x7eff24101990 legacy=0x7eff241a3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:48.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff241a6fc0 con 0x7eff240ffa70 2024-03-12T03:09:48.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff2a0d4700 1 -- 172.21.15.151:0/2270687691 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff241a4fa0 con 0x7eff240ffa70 2024-03-12T03:09:48.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff2a0d4700 1 -- 172.21.15.151:0/2270687691 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff241a5460 con 0x7eff240ffa70 2024-03-12T03:09:48.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7eff14001ba0 con 0x7eff240ffa70 2024-03-12T03:09:48.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.279+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 239+0+0 (unknown 1839991146 0 0) 0x7eff14004d50 con 0x7eff240ffa70 2024-03-12T03:09:48.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.280+0000 7eff2a0d4700 1 -- 172.21.15.151:0/2270687691 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff04005120 con 0x7eff240ffa70 2024-03-12T03:09:48.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.280+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7eff14004f10 con 0x7eff240ffa70 2024-03-12T03:09:48.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.280+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 3179478896 0 0) 0x7eff14047780 con 0x7eff240ffa70 2024-03-12T03:09:48.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.285+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7eff14017400 con 0x7eff240ffa70 2024-03-12T03:09:48.437 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 podman[40465]: 2024-03-12 03:09:48.083965272 +0000 UTC m=+0.019249375 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:09:48.438 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 podman[40465]: 2024-03-12 03:09:48.180746595 +0000 UTC m=+0.116030741 container create 09f920794c6cb263d315fb2f233703155436059e19b2c6ec85a5cd643a7c1392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True) 2024-03-12T03:09:48.438 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 podman[40465]: 2024-03-12 03:09:48.408859844 +0000 UTC m=+0.344143963 container init 09f920794c6cb263d315fb2f233703155436059e19b2c6ec85a5cd643a7c1392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-03-12T03:09:48.438 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 podman[40465]: 2024-03-12 03:09:48.412412621 +0000 UTC m=+0.347696734 container start 09f920794c6cb263d315fb2f233703155436059e19b2c6ec85a5cd643a7c1392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-12T03:09:48.438 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:09:48.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.460+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7eff14014ea0 con 0x7eff240ffa70 2024-03-12T03:09:48.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:48.565+0000 7eff2a0d4700 1 -- 172.21.15.151:0/2270687691 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff04005410 con 0x7eff240ffa70 2024-03-12T03:09:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 2 2024-03-12T03:09:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: pidfile_write: ignore empty --pid-file 2024-03-12T03:09:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: load: jerasure load: lrc 2024-03-12T03:09:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: RocksDB version: 7.9.2 2024-03-12T03:09:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Git sha 0 2024-03-12T03:09:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-12T03:09:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: DB SUMMARY 2024-03-12T03:09:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: DB Session ID: HFQ3GYHSKTHAGH9EX7U1 2024-03-12T03:09:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:09:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:09:48.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-12T03:09:48.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-12T03:09:48.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-03-12T03:09:48.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:09:48.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:09:48.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:09:48.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T03:09:48.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:09:48.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T03:09:48.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.env: 0x56470a6c0b20 2024-03-12T03:09:48.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T03:09:48.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.info_log: 0x56470c03b900 2024-03-12T03:09:48.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.statistics: (nil) 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.use_fsync: 0 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:09:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.db_log_dir: 2024-03-12T03:09:48.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.wal_dir: 2024-03-12T03:09:48.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:09:48.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:09:48.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:09:48.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:09:48.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:09:48.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.write_buffer_manager: 0x56470c00f400 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.unordered_write: 0 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:09:48.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.row_cache: None 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.wal_filter: None 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.wal_compression: 0 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:09:48.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:09:48.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_open_files: -1 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Compression algorithms supported: 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: kZSTD supported: 0 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:09:48.718 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.merge_operator: 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compaction_filter: None 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56470c03ba00) 2024-03-12T03:09:48.719 INFO:journalctl@ceph.mon.c.smithi094.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:09:48.720 INFO:journalctl@ceph.mon.c.smithi094.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:09:48.720 INFO:journalctl@ceph.mon.c.smithi094.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:09:48.720 INFO:journalctl@ceph.mon.c.smithi094.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:09:48.720 INFO:journalctl@ceph.mon.c.smithi094.stdout: index_type: 0 2024-03-12T03:09:48.720 INFO:journalctl@ceph.mon.c.smithi094.stdout: data_block_index_type: 0 2024-03-12T03:09:48.720 INFO:journalctl@ceph.mon.c.smithi094.stdout: index_shortening: 1 2024-03-12T03:09:48.720 INFO:journalctl@ceph.mon.c.smithi094.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:09:48.720 INFO:journalctl@ceph.mon.c.smithi094.stdout: checksum: 4 2024-03-12T03:09:48.720 INFO:journalctl@ceph.mon.c.smithi094.stdout: no_block_cache: 0 2024-03-12T03:09:48.720 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_cache: 0x56470c041090 2024-03-12T03:09:48.720 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:09:48.720 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_cache_options: 2024-03-12T03:09:48.720 INFO:journalctl@ceph.mon.c.smithi094.stdout: capacity : 536870912 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: num_shard_bits : 4 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: strict_capacity_limit : 0 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_cache_compressed: (nil) 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: persistent_cache: (nil) 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_size: 4096 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_size_deviation: 10 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_restart_interval: 16 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: index_block_restart_interval: 1 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: metadata_block_size: 4096 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: partition_filters: 0 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: use_delta_encoding: 1 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: filter_policy: bloomfilter 2024-03-12T03:09:48.721 INFO:journalctl@ceph.mon.c.smithi094.stdout: whole_key_filtering: 1 2024-03-12T03:09:48.722 INFO:journalctl@ceph.mon.c.smithi094.stdout: verify_compression: 0 2024-03-12T03:09:48.722 INFO:journalctl@ceph.mon.c.smithi094.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:09:48.722 INFO:journalctl@ceph.mon.c.smithi094.stdout: format_version: 5 2024-03-12T03:09:48.722 INFO:journalctl@ceph.mon.c.smithi094.stdout: enable_index_compression: 1 2024-03-12T03:09:48.722 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_align: 0 2024-03-12T03:09:48.722 INFO:journalctl@ceph.mon.c.smithi094.stdout: max_auto_readahead_size: 262144 2024-03-12T03:09:48.722 INFO:journalctl@ceph.mon.c.smithi094.stdout: prepopulate_block_cache: 0 2024-03-12T03:09:48.722 INFO:journalctl@ceph.mon.c.smithi094.stdout: initial_auto_readahead_size: 8192 2024-03-12T03:09:48.722 INFO:journalctl@ceph.mon.c.smithi094.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T03:09:48.722 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:09:48.722 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:09:48.722 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compression: NoCompression 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.num_levels: 7 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:09:48.723 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:09:48.724 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:09:48.724 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:09:48.724 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:09:48.724 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:09:48.724 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:09:48.724 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:09:48.724 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:09:48.724 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:09:48.724 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:09:48.724 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:09:48.724 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:09:48.724 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:09:48.724 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T03:09:48.725 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:09:48.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.ttl: 2592000 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:09:48.727 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-12T03:09:48.728 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 937d68d6-c04c-4234-896e-62004feba3b3 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710212988441812, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710212988442191, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1677, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710212988, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "937d68d6-c04c-4234-896e-62004feba3b3", "db_session_id": "HFQ3GYHSKTHAGH9EX7U1", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710212988442272, "job": 1, "event": "recovery_finished"} 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56470c134000 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: DB pointer 0x56470c11e000 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout: ** DB Stats ** 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:09:48.729 INFO:journalctl@ceph.mon.c.smithi094.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:09:48.730 INFO:journalctl@ceph.mon.c.smithi094.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:09:48.730 INFO:journalctl@ceph.mon.c.smithi094.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:09:48.730 INFO:journalctl@ceph.mon.c.smithi094.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:09:48.730 INFO:journalctl@ceph.mon.c.smithi094.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:09:48.730 INFO:journalctl@ceph.mon.c.smithi094.stdout: 2024-03-12T03:09:48.730 INFO:journalctl@ceph.mon.c.smithi094.stdout: ** Compaction Stats [default] ** 2024-03-12T03:09:48.730 INFO:journalctl@ceph.mon.c.smithi094.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:09:48.730 INFO:journalctl@ceph.mon.c.smithi094.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T03:09:48.730 INFO:journalctl@ceph.mon.c.smithi094.stdout: L0 1/0 1.64 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:09:48.730 INFO:journalctl@ceph.mon.c.smithi094.stdout: Sum 1/0 1.64 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:09:48.730 INFO:journalctl@ceph.mon.c.smithi094.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:09:48.730 INFO:journalctl@ceph.mon.c.smithi094.stdout: 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: ** Compaction Stats [default] ** 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:09:48.731 INFO:journalctl@ceph.mon.c.smithi094.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout: Block cache BinnedLRUCache@0x56470c041090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout: 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: using public_addrv v1:172.21.15.94:6790/0 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: starting mon.c rank -1 at public addrs v1:172.21.15.94:6790/0 at bind addrs v1:172.21.15.94:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(???) e0 preinit fsid 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(synchronizing).mds e1 new map 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout: e1 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:09:48.732 INFO:journalctl@ceph.mon.c.smithi094.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout: legacy client fscid: -1 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout: 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout: No filesystems configured 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:48.733 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='client.14180 v1:172.21.15.94:0/370855332' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/209802826' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:48.734 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/209802826' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='client.14184 v1:172.21.15.151:0/694102970' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi094:[v1:172.21.15.94:6789]=a;smithi094:[v1:172.21.15.94:6790]=c;smithi151:[v1:172.21.15.151:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: Saving service mon spec with placement smithi094:[v1:172.21.15.94:6789]=a;smithi094:[v1:172.21.15.94:6790]=c;smithi151:[v1:172.21.15.151:6789]=b;count:3 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:48.735 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: Deploying daemon mon.b on smithi151 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/145405499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.a calling monitor election 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.b calling monitor election 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:48.736 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: monmap e2: 2 mons at {a=v1:172.21.15.94:6789/0,b=v1:172.21.15.151:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: fsmap 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mgrmap e13: y(active, since 42s) 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: overall HEALTH_OK 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: Deploying daemon mon.c on smithi094 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/4249169791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:09:48.737 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:48.738 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:48.738 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 ceph-mon[40484]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-03-12T03:09:48.738 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 bash[40465]: 09f920794c6cb263d315fb2f233703155436059e19b2c6ec85a5cd643a7c1392 2024-03-12T03:09:48.738 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:48 smithi094 systemd[1]: Started Ceph mon.c for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:09:53.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:53.493+0000 7eff217fa700 1 -- 172.21.15.151:0/2270687691 <== mon.0 v1:172.21.15.94:6789/0 11 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1127 (unknown 409899127 0 1605451812) 0x7eff14016d00 con 0x7eff240ffa70 2024-03-12T03:09:53.496 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-12T03:09:53.496 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":3,"fsid":"964142e8-e01d-11ee-95c8-87774f69a715","modified":"2024-03-12T03:09:48.460373Z","created":"2024-03-12T03:08:04.720419Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6790","nonce":0}]},"addr":"172.21.15.94:6790/0","public_addr":"172.21.15.94:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-12T03:09:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:53.496+0000 7eff2a0d4700 1 -- 172.21.15.151:0/2270687691 >> v1:172.21.15.94:6800/1299282424 conn(0x7eff1003a510 legacy=0x7eff1003c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:53.496+0000 7eff2a0d4700 1 -- 172.21.15.151:0/2270687691 >> v1:172.21.15.94:6789/0 conn(0x7eff240ffa70 legacy=0x7eff24101280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:53.496+0000 7eff2a0d4700 1 -- 172.21.15.151:0/2270687691 shutdown_connections 2024-03-12T03:09:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:53.496+0000 7eff2a0d4700 1 -- 172.21.15.151:0/2270687691 >> 172.21.15.151:0/2270687691 conn(0x7eff240fb570 msgr2=0x7eff240fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:53.496+0000 7eff2a0d4700 1 -- 172.21.15.151:0/2270687691 shutdown_connections 2024-03-12T03:09:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:53.496+0000 7eff2a0d4700 1 -- 172.21.15.151:0/2270687691 wait complete. 2024-03-12T03:09:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:dumped monmap epoch 3 2024-03-12T03:09:53.874 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: mon.a calling monitor election 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: mon.b calling monitor election 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/2270687691' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: mon.c calling monitor election 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.875 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-12T03:09:53.876 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: monmap e3: 3 mons at {a=v1:172.21.15.94:6789/0,b=v1:172.21.15.151:6789/0,c=v1:172.21.15.94:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:09:53.876 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: fsmap 2024-03-12T03:09:53.876 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:09:53.876 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: mgrmap e13: y(active, since 51s) 2024-03-12T03:09:53.876 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: overall HEALTH_OK 2024-03-12T03:09:53.876 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:53.876 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:53.876 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:53.876 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:53.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:53.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:09:53.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: mon.a calling monitor election 2024-03-12T03:09:53.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: mon.b calling monitor election 2024-03-12T03:09:53.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:53.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/2270687691' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: mon.c calling monitor election 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: monmap e3: 3 mons at {a=v1:172.21.15.94:6789/0,b=v1:172.21.15.151:6789/0,c=v1:172.21.15.94:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: fsmap 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: mgrmap e13: y(active, since 51s) 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: overall HEALTH_OK 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:53.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:53.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:53.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:53.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:09:53.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: mon.a calling monitor election 2024-03-12T03:09:53.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: mon.b calling monitor election 2024-03-12T03:09:53.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:09:53.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/2270687691' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:09:53.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:53.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: mon.c calling monitor election 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: monmap e3: 3 mons at {a=v1:172.21.15.94:6789/0,b=v1:172.21.15.151:6789/0,c=v1:172.21.15.94:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: fsmap 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: mgrmap e13: y(active, since 51s) 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: overall HEALTH_OK 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:53.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:54.199 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-12T03:09:54.199 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-12T03:09:54.859 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:09:54.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:54 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:09:54.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:54 smithi094 ceph-mon[40484]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:54.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:54 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:54.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:54 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:09:54.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:54 smithi094 ceph-mon[29291]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:54.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:54 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:54 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:09:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:54 smithi151 ceph-mon[32923]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:54 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:09:55.650 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:09:55 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:09:55.459+0000 7f86ac23f700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-12T03:09:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:09:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: Updating smithi094:/etc/ceph/ceph.conf 2024-03-12T03:09:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-12T03:09:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: Updating smithi094:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:09:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:09:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:09:56.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:09:56.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:56.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: Updating smithi094:/etc/ceph/ceph.conf 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: Updating smithi094:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.213 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.213 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.213 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.213 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.213 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:09:56.213 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:09:56.213 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:56.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.450+0000 7f0640563700 1 -- 172.21.15.94:0/2303456368 >> v1:172.21.15.94:6789/0 conn(0x7f0638102890 legacy=0x7f0638102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:56.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.450+0000 7f0640563700 1 -- 172.21.15.94:0/2303456368 shutdown_connections 2024-03-12T03:09:56.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.450+0000 7f0640563700 1 -- 172.21.15.94:0/2303456368 >> 172.21.15.94:0/2303456368 conn(0x7f06380fc0a0 msgr2=0x7f06380fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:56.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.450+0000 7f0640563700 1 -- 172.21.15.94:0/2303456368 shutdown_connections 2024-03-12T03:09:56.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.450+0000 7f0640563700 1 -- 172.21.15.94:0/2303456368 wait complete. 2024-03-12T03:09:56.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.451+0000 7f0640563700 1 Processor -- start 2024-03-12T03:09:56.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.451+0000 7f0640563700 1 -- start start 2024-03-12T03:09:56.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.451+0000 7f0640563700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f06381a65b0 con 0x7f06381a2860 2024-03-12T03:09:56.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.451+0000 7f0640563700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f06381a77d0 con 0x7f0638196ce0 2024-03-12T03:09:56.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.451+0000 7f0640563700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f06381a8a10 con 0x7f0638102890 2024-03-12T03:09:56.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.451+0000 7f063eb00700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f06381a2860 0x7f06381a4cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47810/0 (socket says 172.21.15.94:47810) 2024-03-12T03:09:56.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.451+0000 7f063eb00700 1 -- 172.21.15.94:0/3947128764 learned_addr learned my addr 172.21.15.94:0/3947128764 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:09:56.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556023542 0 0) 0x7f06381a65b0 con 0x7f06381a2860 2024-03-12T03:09:56.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0624003bf0 con 0x7f06381a2860 2024-03-12T03:09:56.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102323738 0 0) 0x7f06381a77d0 con 0x7f0638196ce0 2024-03-12T03:09:56.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06381a65b0 con 0x7f0638196ce0 2024-03-12T03:09:56.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 697438401 0 0) 0x7f06381a65b0 con 0x7f0638196ce0 2024-03-12T03:09:56.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06381a77d0 con 0x7f0638196ce0 2024-03-12T03:09:56.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f0628001810 con 0x7f0638196ce0 2024-03-12T03:09:56.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3623823201 0 0) 0x7f0624003bf0 con 0x7f06381a2860 2024-03-12T03:09:56.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06381a65b0 con 0x7f06381a2860 2024-03-12T03:09:56.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f0630001b70 con 0x7f06381a2860 2024-03-12T03:09:56.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3398890008 0 0) 0x7f06381a65b0 con 0x7f06381a2860 2024-03-12T03:09:56.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 >> v1:172.21.15.94:6790/0 conn(0x7f0638102890 legacy=0x7f06381965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:56.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 >> v1:172.21.15.151:6789/0 conn(0x7f0638196ce0 legacy=0x7f06381a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:56.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06381a9c50 con 0x7f06381a2860 2024-03-12T03:09:56.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.452+0000 7f0640563700 1 -- 172.21.15.94:0/3947128764 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06381a67e0 con 0x7f06381a2860 2024-03-12T03:09:56.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.453+0000 7f0640563700 1 -- 172.21.15.94:0/3947128764 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06381a6ca0 con 0x7f06381a2860 2024-03-12T03:09:56.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.453+0000 7f0640563700 1 -- 172.21.15.94:0/3947128764 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0638063410 con 0x7f06381a2860 2024-03-12T03:09:56.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.454+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f06300051c0 con 0x7f06381a2860 2024-03-12T03:09:56.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.454+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f0630005f70 con 0x7f06381a2860 2024-03-12T03:09:56.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.455+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7f0630011570 con 0x7f06381a2860 2024-03-12T03:09:56.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.455+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 3179478896 0 0) 0x7f06300486b0 con 0x7f06381a2860 2024-03-12T03:09:56.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.459+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f0630017540 con 0x7f06381a2860 2024-03-12T03:09:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: Updating smithi094:/etc/ceph/ceph.conf 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: Updating smithi094:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:09:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:09:56.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:56.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.662+0000 7f0640563700 1 -- 172.21.15.94:0/3947128764 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f063810c340 con 0x7f06381a2860 2024-03-12T03:09:56.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.662+0000 7f062f7fe700 1 -- 172.21.15.94:0/3947128764 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+191 (unknown 2189217731 0 3864097978) 0x7f0630016e40 con 0x7f06381a2860 2024-03-12T03:09:56.665 INFO:teuthology.orchestra.run.smithi094.stdout:# minimal ceph.conf for 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:09:56.665 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2024-03-12T03:09:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:09:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: mon_host = 172.21.15.94:6789/0 172.21.15.151:6789/0 v1:172.21.15.94:6790/0 2024-03-12T03:09:56.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.664+0000 7f0640563700 1 -- 172.21.15.94:0/3947128764 >> v1:172.21.15.94:6800/1299282424 conn(0x7f062403a180 legacy=0x7f062403c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:56.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.664+0000 7f0640563700 1 -- 172.21.15.94:0/3947128764 >> v1:172.21.15.94:6789/0 conn(0x7f06381a2860 legacy=0x7f06381a4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:56.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.664+0000 7f0640563700 1 -- 172.21.15.94:0/3947128764 shutdown_connections 2024-03-12T03:09:56.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.664+0000 7f0640563700 1 -- 172.21.15.94:0/3947128764 >> 172.21.15.94:0/3947128764 conn(0x7f06380fc0a0 msgr2=0x7f06380fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:56.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.665+0000 7f0640563700 1 -- 172.21.15.94:0/3947128764 shutdown_connections 2024-03-12T03:09:56.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:09:56.665+0000 7f0640563700 1 -- 172.21.15.94:0/3947128764 wait complete. 2024-03-12T03:09:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:57 smithi094 ceph-mon[29291]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T03:09:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:57 smithi094 ceph-mon[29291]: Reconfiguring daemon mon.a on smithi094 2024-03-12T03:09:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:57 smithi094 ceph-mon[29291]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3947128764' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:57 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:57 smithi094 ceph-mon[40484]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T03:09:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:57 smithi094 ceph-mon[40484]: Reconfiguring daemon mon.a on smithi094 2024-03-12T03:09:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:57 smithi094 ceph-mon[40484]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3947128764' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:57 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:57 smithi151 ceph-mon[32923]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T03:09:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:57 smithi151 ceph-mon[32923]: Reconfiguring daemon mon.a on smithi094 2024-03-12T03:09:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:57 smithi151 ceph-mon[32923]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3947128764' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:57 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:57.755 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-12T03:09:57.755 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:09:57.755 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T03:09:57.789 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:09:57.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:09:57.858 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-12T03:09:57.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T03:09:57.893 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-12T03:09:57.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:09:57.964 INFO:tasks.cephadm:Adding mgr.y on smithi094 2024-03-12T03:09:57.964 INFO:tasks.cephadm:Adding mgr.x on smithi151 2024-03-12T03:09:57.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch apply mgr '2;smithi094=y;smithi151=x' 2024-03-12T03:09:58.243 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:09:59.275 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:59 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:59.275 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:59 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:59.275 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:59 smithi151 ceph-mon[32923]: Reconfiguring mon.c (monmap changed)... 2024-03-12T03:09:59.276 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:59 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:09:59.276 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:59 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:09:59.276 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:59 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:59.276 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:59 smithi151 ceph-mon[32923]: Reconfiguring daemon mon.c on smithi094 2024-03-12T03:09:59.276 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:09:59 smithi151 ceph-mon[32923]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:59.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[29291]: Reconfiguring mon.c (monmap changed)... 2024-03-12T03:09:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:09:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:09:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[29291]: Reconfiguring daemon mon.c on smithi094 2024-03-12T03:09:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[29291]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:59.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:59.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:09:59.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[40484]: Reconfiguring mon.c (monmap changed)... 2024-03-12T03:09:59.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:09:59.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:09:59.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:09:59.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[40484]: Reconfiguring daemon mon.c on smithi094 2024-03-12T03:09:59.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:09:59 smithi094 ceph-mon[40484]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:09:59.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.765+0000 7f1401c63700 1 -- 172.21.15.151:0/1608545839 >> v1:172.21.15.94:6789/0 conn(0x7f13fc100130 legacy=0x7f13fc100550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:59.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.766+0000 7f1401c63700 1 -- 172.21.15.151:0/1608545839 shutdown_connections 2024-03-12T03:09:59.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.766+0000 7f1401c63700 1 -- 172.21.15.151:0/1608545839 >> 172.21.15.151:0/1608545839 conn(0x7f13fc0fbd30 msgr2=0x7f13fc0fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:09:59.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.766+0000 7f1401c63700 1 -- 172.21.15.151:0/1608545839 shutdown_connections 2024-03-12T03:09:59.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.766+0000 7f1401c63700 1 -- 172.21.15.151:0/1608545839 wait complete. 2024-03-12T03:09:59.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.766+0000 7f1401c63700 1 Processor -- start 2024-03-12T03:09:59.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f1401c63700 1 -- start start 2024-03-12T03:09:59.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f1401c63700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f13fc1a3da0 con 0x7f13fc100130 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f1401c63700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f13fc1a4fe0 con 0x7f13fc1a00c0 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f1401c63700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f13fc1a6220 con 0x7f13fc198bd0 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13fbfff700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f13fc1a00c0 0x7f13fc1a24b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:33838/0 (socket says 172.21.15.151:33838) 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13fbfff700 1 -- 172.21.15.151:0/2930457344 learned_addr learned my addr 172.21.15.151:0/2930457344 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503074070 0 0) 0x7f13fc1a4fe0 con 0x7f13fc1a00c0 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e4003bf0 con 0x7f13fc1a00c0 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419382963 0 0) 0x7f13fc1a3da0 con 0x7f13fc100130 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13fc1a4fe0 con 0x7f13fc100130 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950112366 0 0) 0x7f13fc1a6220 con 0x7f13fc198bd0 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13fc1a3da0 con 0x7f13fc198bd0 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 947616604 0 0) 0x7f13e4003bf0 con 0x7f13fc1a00c0 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13fc1a6220 con 0x7f13fc1a00c0 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f13f40017f0 con 0x7f13fc1a00c0 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3953011084 0 0) 0x7f13fc1a4fe0 con 0x7f13fc100130 2024-03-12T03:09:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13e4003bf0 con 0x7f13fc100130 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.767+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f13ec003c90 con 0x7f13fc100130 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.768+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2028716048 0 0) 0x7f13fc1a3da0 con 0x7f13fc198bd0 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.768+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13fc1a4fe0 con 0x7f13fc198bd0 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.768+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f13f0001750 con 0x7f13fc198bd0 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.768+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3107635689 0 0) 0x7f13fc1a6220 con 0x7f13fc1a00c0 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.768+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 >> v1:172.21.15.94:6790/0 conn(0x7f13fc198bd0 legacy=0x7f13fc19e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.768+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 >> v1:172.21.15.94:6789/0 conn(0x7f13fc100130 legacy=0x7f13fc1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.768+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13fc1a7460 con 0x7f13fc1a00c0 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.768+0000 7f1401c63700 1 -- 172.21.15.151:0/2930457344 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13fc1a3fd0 con 0x7f13fc1a00c0 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.768+0000 7f1401c63700 1 -- 172.21.15.151:0/2930457344 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13fc1a4510 con 0x7f13fc1a00c0 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.768+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f13f4004e40 con 0x7f13fc1a00c0 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.768+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f13f4005d50 con 0x7f13fc1a00c0 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.768+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7f13f4011430 con 0x7f13fc1a00c0 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.769+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 3179478896 0 0) 0x7f13f4048300 con 0x7f13fc1a00c0 2024-03-12T03:09:59.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.769+0000 7f1401c63700 1 -- 172.21.15.151:0/2930457344 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e8005150 con 0x7f13fc1a00c0 2024-03-12T03:09:59.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.774+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f13f4017470 con 0x7f13fc1a00c0 2024-03-12T03:09:59.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.989+0000 7f1401c63700 1 -- 172.21.15.151:0/2930457344 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi094=y;smithi151=x", "target": ["mon-mgr", ""]}) v1 -- 0x7f13e8000bc0 con 0x7f13e403a440 2024-03-12T03:09:59.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.995+0000 7f13f8ff9700 1 -- 172.21.15.151:0/2930457344 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 325935098) 0x7f13e8000bc0 con 0x7f13e403a440 2024-03-12T03:09:59.998 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled mgr update... 2024-03-12T03:10:00.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.997+0000 7f1401c63700 1 -- 172.21.15.151:0/2930457344 >> v1:172.21.15.94:6800/1299282424 conn(0x7f13e403a440 legacy=0x7f13e403c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:00.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.997+0000 7f1401c63700 1 -- 172.21.15.151:0/2930457344 >> v1:172.21.15.151:6789/0 conn(0x7f13fc1a00c0 legacy=0x7f13fc1a24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:00.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.998+0000 7f1401c63700 1 -- 172.21.15.151:0/2930457344 shutdown_connections 2024-03-12T03:10:00.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.998+0000 7f1401c63700 1 -- 172.21.15.151:0/2930457344 >> 172.21.15.151:0/2930457344 conn(0x7f13fc0fbd30 msgr2=0x7f13fc0fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:10:00.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.998+0000 7f1401c63700 1 -- 172.21.15.151:0/2930457344 shutdown_connections 2024-03-12T03:10:00.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:09:59.998+0000 7f1401c63700 1 -- 172.21.15.151:0/2930457344 wait complete. 2024-03-12T03:10:00.523 DEBUG:teuthology.orchestra.run.smithi151:mgr.x> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@mgr.x.service 2024-03-12T03:10:00.525 INFO:tasks.cephadm:Deploying OSDs... 2024-03-12T03:10:00.525 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:10:00.525 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:10:00.544 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T03:10:00.544 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2024-03-12T03:10:00.601 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T03:10:00.601 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:10:00.602 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-12T03:10:00.602 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:10:00.602 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:10:00.602 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-12 03:09:47.905449785 +0000 2024-03-12T03:10:00.602 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-12 03:03:34.827113749 +0000 2024-03-12T03:10:00.602 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-12 03:03:34.827113749 +0000 2024-03-12T03:10:00.602 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-12 03:03:34.827113749 +0000 2024-03-12T03:10:00.602 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T03:10:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-12T03:10:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-12T03:10:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000122463 s, 4.2 MB/s 2024-03-12T03:10:00.672 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T03:10:00.730 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2024-03-12T03:10:00.788 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T03:10:00.788 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:10:00.788 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-12T03:10:00.788 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:10:00.788 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:10:00.788 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-12 03:09:47.906449768 +0000 2024-03-12T03:10:00.788 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-12 03:03:35.153108000 +0000 2024-03-12T03:10:00.788 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-12 03:03:35.153108000 +0000 2024-03-12T03:10:00.788 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-12 03:03:35.153108000 +0000 2024-03-12T03:10:00.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T03:10:00.855 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-12T03:10:00.855 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-12T03:10:00.855 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000128361 s, 4.0 MB/s 2024-03-12T03:10:00.857 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T03:10:00.913 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2024-03-12T03:10:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[29291]: Reconfiguring mon.b (monmap changed)... 2024-03-12T03:10:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:10:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:10:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[29291]: Reconfiguring daemon mon.b on smithi151 2024-03-12T03:10:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[29291]: from='client.24106 v1:172.21.15.151:0/2930457344' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi094=y;smithi151=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:10:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[29291]: Saving service mgr spec with placement smithi094=y;smithi151=x;count:2 2024-03-12T03:10:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[29291]: overall HEALTH_OK 2024-03-12T03:10:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[29291]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[40484]: Reconfiguring mon.b (monmap changed)... 2024-03-12T03:10:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:10:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:10:00.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:00.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[40484]: Reconfiguring daemon mon.b on smithi151 2024-03-12T03:10:00.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[40484]: from='client.24106 v1:172.21.15.151:0/2930457344' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi094=y;smithi151=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:10:00.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[40484]: Saving service mgr spec with placement smithi094=y;smithi151=x;count:2 2024-03-12T03:10:00.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:00.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[40484]: overall HEALTH_OK 2024-03-12T03:10:00.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:00 smithi094 ceph-mon[40484]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:00.968 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T03:10:00.968 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:10:00.969 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-12T03:10:00.969 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:10:00.969 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:10:00.969 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-12 03:09:47.906449768 +0000 2024-03-12T03:10:00.969 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-12 03:03:35.491102038 +0000 2024-03-12T03:10:00.969 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-12 03:03:35.491102038 +0000 2024-03-12T03:10:00.969 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-12 03:03:35.491102038 +0000 2024-03-12T03:10:00.969 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T03:10:01.018 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:00 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:01.018 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:00 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:01.018 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:00 smithi151 ceph-mon[32923]: Reconfiguring mon.b (monmap changed)... 2024-03-12T03:10:01.018 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:00 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:10:01.019 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:00 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:10:01.019 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:00 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:01.019 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:00 smithi151 ceph-mon[32923]: Reconfiguring daemon mon.b on smithi151 2024-03-12T03:10:01.019 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:00 smithi151 ceph-mon[32923]: from='client.24106 v1:172.21.15.151:0/2930457344' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi094=y;smithi151=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:10:01.019 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:00 smithi151 ceph-mon[32923]: Saving service mgr spec with placement smithi094=y;smithi151=x;count:2 2024-03-12T03:10:01.019 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:00 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:01.019 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:00 smithi151 ceph-mon[32923]: overall HEALTH_OK 2024-03-12T03:10:01.019 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:00 smithi151 ceph-mon[32923]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:01.035 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-12T03:10:01.035 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-12T03:10:01.035 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000116942 s, 4.4 MB/s 2024-03-12T03:10:01.036 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T03:10:01.092 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2024-03-12T03:10:01.147 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T03:10:01.148 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:10:01.148 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-12T03:10:01.148 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:10:01.148 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:10:01.148 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-12 03:09:47.906449768 +0000 2024-03-12T03:10:01.148 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-12 03:03:35.815096325 +0000 2024-03-12T03:10:01.148 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-12 03:03:35.815096325 +0000 2024-03-12T03:10:01.148 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-12 03:03:35.815096325 +0000 2024-03-12T03:10:01.148 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T03:10:01.213 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-12T03:10:01.213 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-12T03:10:01.214 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000176929 s, 2.9 MB/s 2024-03-12T03:10:01.215 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T03:10:01.272 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-12T03:10:01.272 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:10:01.288 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T03:10:01.288 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2024-03-12T03:10:01.343 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T03:10:01.343 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:10:01.343 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-12T03:10:01.343 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:10:01.343 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:10:01.343 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-12 03:09:38.658641642 +0000 2024-03-12T03:10:01.343 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-12 03:03:34.843345776 +0000 2024-03-12T03:10:01.343 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-12 03:03:34.843345776 +0000 2024-03-12T03:10:01.343 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-12 03:03:34.843345776 +0000 2024-03-12T03:10:01.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T03:10:01.409 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-12T03:10:01.409 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-12T03:10:01.409 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000214957 s, 2.4 MB/s 2024-03-12T03:10:01.411 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T03:10:01.467 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2024-03-12T03:10:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T03:10:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:10:01.522 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-12T03:10:01.522 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:10:01.522 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:10:01.522 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-12 03:09:38.658641642 +0000 2024-03-12T03:10:01.522 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-12 03:03:35.180341770 +0000 2024-03-12T03:10:01.522 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-12 03:03:35.180341770 +0000 2024-03-12T03:10:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-12 03:03:35.180341770 +0000 2024-03-12T03:10:01.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T03:10:01.587 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-12T03:10:01.588 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-12T03:10:01.588 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000244976 s, 2.1 MB/s 2024-03-12T03:10:01.589 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T03:10:01.644 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2024-03-12T03:10:01.700 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T03:10:01.700 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:10:01.700 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-12T03:10:01.700 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:10:01.701 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:10:01.701 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-12 03:09:38.659641630 +0000 2024-03-12T03:10:01.701 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-12 03:03:35.517337765 +0000 2024-03-12T03:10:01.701 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-12 03:03:35.517337765 +0000 2024-03-12T03:10:01.701 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-12 03:03:35.517337765 +0000 2024-03-12T03:10:01.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T03:10:01.766 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-12T03:10:01.766 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-12T03:10:01.766 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.00020243 s, 2.5 MB/s 2024-03-12T03:10:01.767 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T03:10:01.824 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2024-03-12T03:10:01.879 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T03:10:01.880 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:10:01.880 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-12T03:10:01.880 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:10:01.880 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:10:01.880 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-12 03:09:38.659641630 +0000 2024-03-12T03:10:01.880 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-12 03:03:35.868333593 +0000 2024-03-12T03:10:01.880 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-12 03:03:35.868333593 +0000 2024-03-12T03:10:01.880 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-12 03:03:35.868333593 +0000 2024-03-12T03:10:01.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T03:10:01.946 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-12T03:10:01.947 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-12T03:10:01.947 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000196828 s, 2.6 MB/s 2024-03-12T03:10:01.948 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T03:10:02.003 INFO:tasks.cephadm:Deploying osd.0 on smithi094 with /dev/vg_nvme/lv_4... 2024-03-12T03:10:02.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T03:10:02.245 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:10:02.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:10:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:10:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:10:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[40484]: Deploying daemon mgr.x on smithi151 2024-03-12T03:10:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[40484]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:02.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:02.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:02.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:02.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:02.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:02.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:02.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:10:02.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:10:02.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:10:02.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:02.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[29291]: Deploying daemon mgr.x on smithi151 2024-03-12T03:10:02.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:02 smithi094 ceph-mon[29291]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:02.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:02.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:02.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:02.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:02.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:02.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:10:02.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:10:02.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:10:02.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:02.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:02 smithi151 ceph-mon[32923]: Deploying daemon mgr.x on smithi151 2024-03-12T03:10:02.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:02 smithi151 ceph-mon[32923]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:03.965 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:03 smithi151 podman[34754]: 2024-03-12 03:10:03.886818572 +0000 UTC m=+0.137502122 container create 804d02ce0a6662750972429604f8fce21d6c5b72ddc3dbe0117990afa8000884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-12T03:10:04.235 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T03:10:04.235 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T03:10:04.235 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-03-12T03:10:04.235 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-03-12T03:10:04.235 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00876732 s, 1.2 GB/s 2024-03-12T03:10:04.236 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-03-12T03:10:04.464 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:04 smithi151 podman[34754]: 2024-03-12 03:10:04.11414497 +0000 UTC m=+0.364828485 container init 804d02ce0a6662750972429604f8fce21d6c5b72ddc3dbe0117990afa8000884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-12T03:10:04.464 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:04 smithi151 podman[34754]: 2024-03-12 03:10:04.116739724 +0000 UTC m=+0.367423239 container start 804d02ce0a6662750972429604f8fce21d6c5b72ddc3dbe0117990afa8000884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x, GIT_CLEAN=True, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-12T03:10:04.465 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:04 smithi151 bash[34754]: 804d02ce0a6662750972429604f8fce21d6c5b72ddc3dbe0117990afa8000884 2024-03-12T03:10:04.465 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:04 smithi151 systemd[1]: Started Ceph mgr.x for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:10:04.777 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:04 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:04.677+0000 7f2128ebb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:10:04.777 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:04 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:04.774+0000 7f2128ebb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:10:04.863 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi094:vg_nvme/lv_4 2024-03-12T03:10:05.098 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:10:05.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:05 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:05.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:05 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:05.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:05 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:05.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:05 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:05.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:05 smithi151 ceph-mon[32923]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:05.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:05 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:05.495 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:05 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:05.495 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:05 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:05.495 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:05 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:05.495 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:05 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:05.495 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:05 smithi094 ceph-mon[40484]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:05.495 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:05 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:05.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:05 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:05.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:05 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:05.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:05 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:05.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:05 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:05.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:05 smithi094 ceph-mon[29291]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:05.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:05 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:05.720 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:05 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:05.462+0000 7f2128ebb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:10:05.721 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:05 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:05.569+0000 7f2128ebb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:10:06.182 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:05 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:05.718+0000 7f2128ebb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:10:06.182 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:05 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:05.817+0000 7f2128ebb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:10:06.182 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:05 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:05.905+0000 7f2128ebb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:10:06.464 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:06.179+0000 7f2128ebb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:10:06.464 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:06.291+0000 7f2128ebb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:10:06.464 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:06.378+0000 7f2128ebb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:10:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[29291]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T03:10:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:10:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:10:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[29291]: Reconfiguring daemon mgr.y on smithi094 2024-03-12T03:10:06.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[29291]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:06.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:06.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:06.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:06.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:06.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:06.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[40484]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T03:10:06.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:10:06.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:10:06.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:06.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[40484]: Reconfiguring daemon mgr.y on smithi094 2024-03-12T03:10:06.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:06 smithi094 ceph-mon[40484]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:06.964 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:06.475+0000 7f2128ebb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:10:06.964 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:06.656+0000 7f2128ebb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:10:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:06.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:06.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:06.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:06.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:06.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-mon[32923]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T03:10:06.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:10:06.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:10:06.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:06.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-mon[32923]: Reconfiguring daemon mgr.y on smithi094 2024-03-12T03:10:06.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:06 smithi151 ceph-mon[32923]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:07.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.429+0000 7fbdf906b700 1 -- 172.21.15.94:0/648540424 >> v1:172.21.15.94:6790/0 conn(0x7fbdf4100260 legacy=0x7fbdf4100680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:07.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.429+0000 7fbdf906b700 1 -- 172.21.15.94:0/648540424 shutdown_connections 2024-03-12T03:10:07.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.429+0000 7fbdf906b700 1 -- 172.21.15.94:0/648540424 >> 172.21.15.94:0/648540424 conn(0x7fbdf40fb800 msgr2=0x7fbdf40fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:10:07.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.429+0000 7fbdf906b700 1 -- 172.21.15.94:0/648540424 shutdown_connections 2024-03-12T03:10:07.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.429+0000 7fbdf906b700 1 -- 172.21.15.94:0/648540424 wait complete. 2024-03-12T03:10:07.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.429+0000 7fbdf906b700 1 Processor -- start 2024-03-12T03:10:07.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf906b700 1 -- start start 2024-03-12T03:10:07.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf906b700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbdf4197b10 con 0x7fbdf4100260 2024-03-12T03:10:07.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf906b700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbdf41a3300 con 0x7fbdf4104690 2024-03-12T03:10:07.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf906b700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbdf41a44e0 con 0x7fbdf41083d0 2024-03-12T03:10:07.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf2d9d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbdf4100260 0x7fbdf4196fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47242/0 (socket says 172.21.15.94:47242) 2024-03-12T03:10:07.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf2d9d700 1 -- 172.21.15.94:0/3268068142 learned_addr learned my addr 172.21.15.94:0/3268068142 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:10:07.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176138337 0 0) 0x7fbdf41a44e0 con 0x7fbdf41083d0 2024-03-12T03:10:07.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde0006350 con 0x7fbdf41083d0 2024-03-12T03:10:07.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996980321 0 0) 0x7fbdf4197b10 con 0x7fbdf4100260 2024-03-12T03:10:07.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf41a44e0 con 0x7fbdf4100260 2024-03-12T03:10:07.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543155858 0 0) 0x7fbdf41a3300 con 0x7fbdf4104690 2024-03-12T03:10:07.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf4197b10 con 0x7fbdf4104690 2024-03-12T03:10:07.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 707593404 0 0) 0x7fbdf41a44e0 con 0x7fbdf4100260 2024-03-12T03:10:07.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdf41a3300 con 0x7fbdf4100260 2024-03-12T03:10:07.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3877909737 0 0) 0x7fbde0006350 con 0x7fbdf41083d0 2024-03-12T03:10:07.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdf41a44e0 con 0x7fbdf41083d0 2024-03-12T03:10:07.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.430+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fbde4004420 con 0x7fbdf4100260 2024-03-12T03:10:07.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.431+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4141671095 0 0) 0x7fbdf4197b10 con 0x7fbdf4104690 2024-03-12T03:10:07.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.431+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbde0006350 con 0x7fbdf4104690 2024-03-12T03:10:07.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.431+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fbdec001bb0 con 0x7fbdf41083d0 2024-03-12T03:10:07.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.431+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fbddc001890 con 0x7fbdf4104690 2024-03-12T03:10:07.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.431+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1861203648 0 0) 0x7fbdf41a3300 con 0x7fbdf4100260 2024-03-12T03:10:07.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.431+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 >> v1:172.21.15.94:6790/0 conn(0x7fbdf41083d0 legacy=0x7fbdf41a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:07.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.431+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 >> v1:172.21.15.151:6789/0 conn(0x7fbdf4104690 legacy=0x7fbdf419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:07.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.431+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf41a5700 con 0x7fbdf4100260 2024-03-12T03:10:07.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.431+0000 7fbdf906b700 1 -- 172.21.15.94:0/3268068142 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf41a2350 con 0x7fbdf4100260 2024-03-12T03:10:07.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.431+0000 7fbdf906b700 1 -- 172.21.15.94:0/3268068142 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf41a2890 con 0x7fbdf4100260 2024-03-12T03:10:07.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.431+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fbde4001b90 con 0x7fbdf4100260 2024-03-12T03:10:07.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.431+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fbde4004f30 con 0x7fbdf4100260 2024-03-12T03:10:07.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.432+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 13) v1 ==== 46127+0+0 (unknown 2088828388 0 0) 0x7fbde4005170 con 0x7fbdf4100260 2024-03-12T03:10:07.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.432+0000 7fbdf906b700 1 -- 172.21.15.94:0/3268068142 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd4005150 con 0x7fbdf4100260 2024-03-12T03:10:07.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.432+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 3179478896 0 0) 0x7fbde40010e0 con 0x7fbdf4100260 2024-03-12T03:10:07.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.437+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fbde4017400 con 0x7fbdf4100260 2024-03-12T03:10:07.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:07.645+0000 7fbdf906b700 1 -- 172.21.15.94:0/3268068142 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdd4000bc0 con 0x7fbde003a1a0 2024-03-12T03:10:07.964 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:07.626+0000 7f2128ebb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:10:07.964 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:07.756+0000 7f2128ebb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:10:08.372 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:08.087+0000 7f2128ebb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:10:08.372 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:08.177+0000 7f2128ebb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:10:08.372 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:08.369+0000 7f2128ebb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:10:08.602 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:08.602 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:10:08.602 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:10:08.602 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:08.602 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:08.602 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:08.602 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:08.602 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:08.603 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:10:08.603 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:10:08.603 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:08.603 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:08.603 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:08.603 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:08 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:08.714 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:08.554+0000 7f2128ebb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:10:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:08 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:08 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:10:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:08 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:10:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:08 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:08 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:08 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:08 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:09.102 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:08.834+0000 7f2128ebb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:10:09.102 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:09.099+0000 7f2128ebb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:10:09.383 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:09.333+0000 7f2128ebb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:10:09.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:09 smithi094 ceph-mon[29291]: from='client.14217 v1:172.21.15.94:0/3268068142' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:10:09.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:09 smithi094 ceph-mon[29291]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:09 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:09 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:09 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:09.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:09 smithi094 ceph-mon[40484]: from='client.14217 v1:172.21.15.94:0/3268068142' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:10:09.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:09 smithi094 ceph-mon[40484]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:09.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:09 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:09.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:09 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:09.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:09 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:09 smithi151 ceph-mon[32923]: from='client.14217 v1:172.21.15.94:0/3268068142' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:10:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:09 smithi151 ceph-mon[32923]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:09 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:09 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:09 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:10.391 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:10 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:10.131+0000 7f2128ebb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:10:10.702 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/796232081' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "003c53a3-baa3-4abc-bbed-e8a0c0db7ab5"}]: dispatch 2024-03-12T03:10:10.703 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/796232081' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "003c53a3-baa3-4abc-bbed-e8a0c0db7ab5"}]': finished 2024-03-12T03:10:10.703 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:10 smithi094 ceph-mon[29291]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T03:10:10.703 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:10 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:10.703 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/796232081' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "003c53a3-baa3-4abc-bbed-e8a0c0db7ab5"}]: dispatch 2024-03-12T03:10:10.703 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/796232081' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "003c53a3-baa3-4abc-bbed-e8a0c0db7ab5"}]': finished 2024-03-12T03:10:10.703 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:10 smithi094 ceph-mon[40484]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T03:10:10.703 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:10 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:10.706 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/796232081' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "003c53a3-baa3-4abc-bbed-e8a0c0db7ab5"}]: dispatch 2024-03-12T03:10:10.706 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/796232081' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "003c53a3-baa3-4abc-bbed-e8a0c0db7ab5"}]': finished 2024-03-12T03:10:10.706 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:10 smithi151 ceph-mon[32923]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T03:10:10.706 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:10 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:10.964 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:10 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:10.703+0000 7f2128ebb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:10:10.964 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:10 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:10.800+0000 7f2128ebb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:10:11.391 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:10:11 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:10:11.015+0000 7f2128ebb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:10:11.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:11.396+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mgrmap(e 14) v1 ==== 91663+0+0 (unknown 3714127607 0 0) 0x7fbde4046db0 con 0x7fbdf4100260 2024-03-12T03:10:11.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[40484]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:11.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4040164727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:10:11.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[40484]: Standby manager daemon x started 2024-03-12T03:10:11.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[40484]: from='mgr.? v1:172.21.15.151:0/1576546515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:10:11.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[40484]: from='mgr.? v1:172.21.15.151:0/1576546515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:10:11.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[40484]: from='mgr.? v1:172.21.15.151:0/1576546515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:10:11.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[40484]: from='mgr.? v1:172.21.15.151:0/1576546515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:10:11.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[29291]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:11.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4040164727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:10:11.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[29291]: Standby manager daemon x started 2024-03-12T03:10:11.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[29291]: from='mgr.? v1:172.21.15.151:0/1576546515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:10:11.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[29291]: from='mgr.? v1:172.21.15.151:0/1576546515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:10:11.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[29291]: from='mgr.? v1:172.21.15.151:0/1576546515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:10:11.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:11 smithi094 ceph-mon[29291]: from='mgr.? v1:172.21.15.151:0/1576546515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:10:11.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:11 smithi151 ceph-mon[32923]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:11.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4040164727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:10:11.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:11 smithi151 ceph-mon[32923]: Standby manager daemon x started 2024-03-12T03:10:11.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:11 smithi151 ceph-mon[32923]: from='mgr.? v1:172.21.15.151:0/1576546515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:10:11.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:11 smithi151 ceph-mon[32923]: from='mgr.? v1:172.21.15.151:0/1576546515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:10:11.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:11 smithi151 ceph-mon[32923]: from='mgr.? v1:172.21.15.151:0/1576546515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:10:11.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:11 smithi151 ceph-mon[32923]: from='mgr.? v1:172.21.15.151:0/1576546515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:10:12.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:12 smithi094 ceph-mon[29291]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-12T03:10:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:12 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:10:12.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:12 smithi094 ceph-mon[40484]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-12T03:10:12.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:12 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:10:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:12 smithi151 ceph-mon[32923]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-12T03:10:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:12 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:10:13.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:13 smithi094 ceph-mon[29291]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:13 smithi094 ceph-mon[40484]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:13.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:13 smithi151 ceph-mon[32923]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:15.467 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:15 smithi094 ceph-mon[29291]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:15.468 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:15 smithi094 ceph-mon[40484]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:15.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:15 smithi151 ceph-mon[32923]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:17.696 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:17 smithi094 ceph-mon[29291]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:17.696 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:17 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:10:17.696 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:17 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:17.697 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:17 smithi094 ceph-mon[40484]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:17.697 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:17 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:10:17.697 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:17 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:17 smithi151 ceph-mon[32923]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:17 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:10:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:17 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:18.450 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:18 smithi094 ceph-mon[29291]: Deploying daemon osd.0 on smithi094 2024-03-12T03:10:18.450 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:18 smithi094 ceph-mon[40484]: Deploying daemon osd.0 on smithi094 2024-03-12T03:10:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:18 smithi151 ceph-mon[32923]: Deploying daemon osd.0 on smithi094 2024-03-12T03:10:19.670 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:19 smithi094 ceph-mon[29291]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:19.671 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:19 smithi094 ceph-mon[40484]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:19 smithi151 ceph-mon[32923]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:21.636 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:21 smithi094 ceph-mon[29291]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:21.636 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:21 smithi094 ceph-mon[40484]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:21.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:21 smithi151 ceph-mon[32923]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:23.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:23 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:23.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:23 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:23.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:23 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:23.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:23 smithi094 ceph-mon[29291]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:23 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:23 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:23 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:23 smithi094 ceph-mon[40484]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:23.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:23 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:23.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:23 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:23.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:23 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:23.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:23 smithi151 ceph-mon[32923]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:25 smithi094 ceph-mon[29291]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:25 smithi094 ceph-mon[29291]: from='osd.0 v1:172.21.15.94:6801/2935792501' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:10:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:25 smithi094 ceph-mon[29291]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:10:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:25 smithi094 ceph-mon[40484]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:25 smithi094 ceph-mon[40484]: from='osd.0 v1:172.21.15.94:6801/2935792501' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:10:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:25 smithi094 ceph-mon[40484]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:10:25.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:25 smithi151 ceph-mon[32923]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:25.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:25 smithi151 ceph-mon[32923]: from='osd.0 v1:172.21.15.94:6801/2935792501' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:10:25.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:25 smithi151 ceph-mon[32923]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:10:25.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:25.900+0000 7fbdf0d99700 1 -- 172.21.15.94:0/3268068142 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 2840496693) 0x7fbdd4000bc0 con 0x7fbde003a1a0 2024-03-12T03:10:25.903 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 0 on host 'smithi094' 2024-03-12T03:10:25.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:25.904+0000 7fbdf906b700 1 -- 172.21.15.94:0/3268068142 >> v1:172.21.15.94:6800/1299282424 conn(0x7fbde003a1a0 legacy=0x7fbde003c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:25.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:25.904+0000 7fbdf906b700 1 -- 172.21.15.94:0/3268068142 >> v1:172.21.15.94:6789/0 conn(0x7fbdf4100260 legacy=0x7fbdf4196fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:25.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:25.904+0000 7fbdf906b700 1 -- 172.21.15.94:0/3268068142 shutdown_connections 2024-03-12T03:10:25.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:25.904+0000 7fbdf906b700 1 -- 172.21.15.94:0/3268068142 >> 172.21.15.94:0/3268068142 conn(0x7fbdf40fb800 msgr2=0x7fbdf4107aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:10:25.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:25.904+0000 7fbdf906b700 1 -- 172.21.15.94:0/3268068142 shutdown_connections 2024-03-12T03:10:25.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:25.904+0000 7fbdf906b700 1 -- 172.21.15.94:0/3268068142 wait complete. 2024-03-12T03:10:26.525 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[29291]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:10:26.525 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[29291]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:10:26.525 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:26.525 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[29291]: from='osd.0 v1:172.21.15.94:6801/2935792501' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:10:26.526 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[29291]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:10:26.526 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:26.526 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:26.526 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:26.526 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:26.526 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[40484]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:10:26.526 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[40484]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:10:26.526 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:26.526 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[40484]: from='osd.0 v1:172.21.15.94:6801/2935792501' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:10:26.526 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[40484]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:10:26.527 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:26.527 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:26.527 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:26.527 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:26.558 DEBUG:teuthology.orchestra.run.smithi094:osd.0> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.0.service 2024-03-12T03:10:26.561 INFO:tasks.cephadm:Deploying osd.1 on smithi094 with /dev/vg_nvme/lv_3... 2024-03-12T03:10:26.561 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T03:10:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:26 smithi151 ceph-mon[32923]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:10:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:26 smithi151 ceph-mon[32923]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:10:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:26 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:26 smithi151 ceph-mon[32923]: from='osd.0 v1:172.21.15.94:6801/2935792501' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:10:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:26 smithi151 ceph-mon[32923]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:10:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:26 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:26 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:26 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:26 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:26.848 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0[44487]: 2024-03-12T03:10:26.672+0000 7f3c12736700 -1 osd.0 0 waiting for initial osdmap 2024-03-12T03:10:26.848 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:10:26 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0[44487]: 2024-03-12T03:10:26.675+0000 7f3c0a521700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:10:27.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[40484]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:27.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[40484]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-03-12T03:10:27.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[40484]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:10:27.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:27.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:27.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[40484]: from='osd.0 ' entity='osd.0' 2024-03-12T03:10:27.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:27.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[29291]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:27.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[29291]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-03-12T03:10:27.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[29291]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:10:27.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:27.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:27.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[29291]: from='osd.0 ' entity='osd.0' 2024-03-12T03:10:27.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:27.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:27 smithi151 ceph-mon[32923]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:27.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:27 smithi151 ceph-mon[32923]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-03-12T03:10:27.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:27 smithi151 ceph-mon[32923]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:10:27.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:27.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:27.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:27 smithi151 ceph-mon[32923]: from='osd.0 ' entity='osd.0' 2024-03-12T03:10:27.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:27.995 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:10:28.858 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:28 smithi094 ceph-mon[29291]: purged_snaps scrub starts 2024-03-12T03:10:28.858 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:28 smithi094 ceph-mon[29291]: purged_snaps scrub ok 2024-03-12T03:10:28.858 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:28 smithi094 ceph-mon[29291]: osd.0 v1:172.21.15.94:6801/2935792501 boot 2024-03-12T03:10:28.859 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:28 smithi094 ceph-mon[29291]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T03:10:28.859 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:28 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:28.859 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:28 smithi094 ceph-mon[29291]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:28.859 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:28 smithi094 ceph-mon[40484]: purged_snaps scrub starts 2024-03-12T03:10:28.859 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:28 smithi094 ceph-mon[40484]: purged_snaps scrub ok 2024-03-12T03:10:28.859 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:28 smithi094 ceph-mon[40484]: osd.0 v1:172.21.15.94:6801/2935792501 boot 2024-03-12T03:10:28.859 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:28 smithi094 ceph-mon[40484]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T03:10:28.859 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:28 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:28.859 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:28 smithi094 ceph-mon[40484]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:28 smithi151 ceph-mon[32923]: purged_snaps scrub starts 2024-03-12T03:10:28.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:28 smithi151 ceph-mon[32923]: purged_snaps scrub ok 2024-03-12T03:10:28.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:28 smithi151 ceph-mon[32923]: osd.0 v1:172.21.15.94:6801/2935792501 boot 2024-03-12T03:10:28.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:28 smithi151 ceph-mon[32923]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T03:10:28.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:28 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:10:28.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:28 smithi151 ceph-mon[32923]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:10:29.798 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:29 smithi094 ceph-mon[40484]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:10:29.798 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:29 smithi094 ceph-mon[29291]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:10:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:29 smithi151 ceph-mon[32923]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:10:31.135 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T03:10:31.135 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T03:10:31.135 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-03-12T03:10:31.135 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-03-12T03:10:31.135 INFO:teuthology.orchestra.run.smithi094.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00842673 s, 1.2 GB/s 2024-03-12T03:10:31.135 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-03-12T03:10:31.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: Detected new or changed devices on smithi094 2024-03-12T03:10:31.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:10:31.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: Adjusting osd_memory_target on smithi094 to 16064M 2024-03-12T03:10:31.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:31.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:31.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:31.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:31.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:31.136 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:31.137 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.137 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: Detected new or changed devices on smithi094 2024-03-12T03:10:31.137 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.137 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.137 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:10:31.137 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: Adjusting osd_memory_target on smithi094 to 16064M 2024-03-12T03:10:31.137 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.137 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:31.137 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:31.137 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.137 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:31.138 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:31.138 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:31.138 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:31.138 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:30 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: Detected new or changed devices on smithi094 2024-03-12T03:10:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:10:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: Adjusting osd_memory_target on smithi094 to 16064M 2024-03-12T03:10:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:30 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:31.764 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi094:vg_nvme/lv_3 2024-03-12T03:10:32.001 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:10:33.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.564+0000 7f597a561700 1 -- 172.21.15.94:0/20439872 >> v1:172.21.15.94:6789/0 conn(0x7f5974075d90 legacy=0x7f59740741f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:33.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.564+0000 7f597a561700 1 -- 172.21.15.94:0/20439872 shutdown_connections 2024-03-12T03:10:33.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.565+0000 7f597a561700 1 -- 172.21.15.94:0/20439872 >> 172.21.15.94:0/20439872 conn(0x7f59740fd190 msgr2=0x7f59740ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:10:33.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.565+0000 7f597a561700 1 -- 172.21.15.94:0/20439872 shutdown_connections 2024-03-12T03:10:33.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.565+0000 7f597a561700 1 -- 172.21.15.94:0/20439872 wait complete. 2024-03-12T03:10:33.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.565+0000 7f597a561700 1 Processor -- start 2024-03-12T03:10:33.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.565+0000 7f597a561700 1 -- start start 2024-03-12T03:10:33.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.565+0000 7f597a561700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5974199dd0 con 0x7f597410c190 2024-03-12T03:10:33.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.565+0000 7f597a561700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f59741a55c0 con 0x7f59740749f0 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.565+0000 7f597a561700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f59741a67a0 con 0x7f5974075d90 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f5978afe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f597410c190 0x7f59741a2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:48008/0 (socket says 172.21.15.94:48008) 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59737fe700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f5974075d90 0x7f597419f760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:44930/0 (socket says 172.21.15.94:44930) 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f5978afe700 1 -- 172.21.15.94:0/1741235534 learned_addr learned my addr 172.21.15.94:0/1741235534 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047696053 0 0) 0x7f59741a67a0 con 0x7f5974075d90 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f595c0035b0 con 0x7f5974075d90 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154012939 0 0) 0x7f5974199dd0 con 0x7f597410c190 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59741a67a0 con 0x7f597410c190 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257971720 0 0) 0x7f59741a55c0 con 0x7f59740749f0 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5974199dd0 con 0x7f59740749f0 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763208986 0 0) 0x7f595c0035b0 con 0x7f5974075d90 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59741a55c0 con 0x7f5974075d90 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f59680018b0 con 0x7f5974075d90 2024-03-12T03:10:33.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1486432491 0 0) 0x7f59741a67a0 con 0x7f597410c190 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f595c0035b0 con 0x7f597410c190 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f596c001c10 con 0x7f597410c190 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 274723549 0 0) 0x7f5974199dd0 con 0x7f59740749f0 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59741a67a0 con 0x7f59740749f0 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f5964004430 con 0x7f59740749f0 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 50965759 0 0) 0x7f59741a55c0 con 0x7f5974075d90 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 >> v1:172.21.15.151:6789/0 conn(0x7f59740749f0 legacy=0x7f5974199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 >> v1:172.21.15.94:6789/0 conn(0x7f597410c190 legacy=0x7f59741a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59741a79c0 con 0x7f5974075d90 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f597a561700 1 -- 172.21.15.94:0/1741235534 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59741a4610 con 0x7f5974075d90 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.566+0000 7f597a561700 1 -- 172.21.15.94:0/1741235534 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59741a4b50 con 0x7f5974075d90 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.567+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f5968003b30 con 0x7f5974075d90 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.567+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f5968005c90 con 0x7f5974075d90 2024-03-12T03:10:33.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.567+0000 7f597a561700 1 -- 172.21.15.94:0/1741235534 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5960005150 con 0x7f5974075d90 2024-03-12T03:10:33.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.568+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 14) v1 ==== 91663+0+0 (unknown 3714127607 0 0) 0x7f596801c510 con 0x7f5974075d90 2024-03-12T03:10:33.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.568+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1541+0+0 (unknown 2797028085 0 0) 0x7f5968087c60 con 0x7f5974075d90 2024-03-12T03:10:33.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.573+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f5968056850 con 0x7f5974075d90 2024-03-12T03:10:33.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:33 smithi094 ceph-mon[29291]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:33.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:33 smithi094 ceph-mon[40484]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:33.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:33 smithi151 ceph-mon[32923]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:33.777+0000 7f597a561700 1 -- 172.21.15.94:0/1741235534 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5960000bc0 con 0x7f595c06eb70 2024-03-12T03:10:34.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:10:34.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:10:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:10:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:10:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:34.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:10:34.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:10:34.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:35.617 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:35 smithi094 ceph-mon[29291]: from='client.34139 v1:172.21.15.94:0/1741235534' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:10:35.617 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:35 smithi094 ceph-mon[29291]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:35.617 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:35 smithi094 ceph-mon[40484]: from='client.34139 v1:172.21.15.94:0/1741235534' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:10:35.618 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:35 smithi094 ceph-mon[40484]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:35 smithi151 ceph-mon[32923]: from='client.34139 v1:172.21.15.94:0/1741235534' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:10:35.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:35 smithi151 ceph-mon[32923]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[29291]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2117717786' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c8e6b7-d57f-492b-bc40-9f46769a1c0a"}]: dispatch 2024-03-12T03:10:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[29291]: from='client.34145 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c8e6b7-d57f-492b-bc40-9f46769a1c0a"}]: dispatch 2024-03-12T03:10:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[29291]: from='client.34145 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91c8e6b7-d57f-492b-bc40-9f46769a1c0a"}]': finished 2024-03-12T03:10:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[29291]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T03:10:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:37.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3742735134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:10:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[40484]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2117717786' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c8e6b7-d57f-492b-bc40-9f46769a1c0a"}]: dispatch 2024-03-12T03:10:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[40484]: from='client.34145 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c8e6b7-d57f-492b-bc40-9f46769a1c0a"}]: dispatch 2024-03-12T03:10:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[40484]: from='client.34145 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91c8e6b7-d57f-492b-bc40-9f46769a1c0a"}]': finished 2024-03-12T03:10:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[40484]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T03:10:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:37.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3742735134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:10:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:37 smithi151 ceph-mon[32923]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2117717786' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c8e6b7-d57f-492b-bc40-9f46769a1c0a"}]: dispatch 2024-03-12T03:10:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:37 smithi151 ceph-mon[32923]: from='client.34145 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c8e6b7-d57f-492b-bc40-9f46769a1c0a"}]: dispatch 2024-03-12T03:10:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:37 smithi151 ceph-mon[32923]: from='client.34145 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91c8e6b7-d57f-492b-bc40-9f46769a1c0a"}]': finished 2024-03-12T03:10:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:37 smithi151 ceph-mon[32923]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T03:10:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3742735134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:10:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:39 smithi094 ceph-mon[29291]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:39 smithi094 ceph-mon[40484]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:39 smithi151 ceph-mon[32923]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:41.699 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:41 smithi094 ceph-mon[29291]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:41.699 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:41 smithi094 ceph-mon[40484]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:41 smithi151 ceph-mon[32923]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:43.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:43 smithi094 ceph-mon[29291]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:43 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:10:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:43 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:43.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:43 smithi094 ceph-mon[40484]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:43.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:43 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:10:43.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:43 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:43 smithi151 ceph-mon[32923]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:43 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:10:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:43 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:44.557 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:44 smithi094 ceph-mon[29291]: Deploying daemon osd.1 on smithi094 2024-03-12T03:10:44.557 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:44 smithi094 ceph-mon[40484]: Deploying daemon osd.1 on smithi094 2024-03-12T03:10:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:44 smithi151 ceph-mon[32923]: Deploying daemon osd.1 on smithi094 2024-03-12T03:10:45.484 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:45 smithi094 ceph-mon[29291]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:45.485 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:45 smithi094 ceph-mon[40484]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:45.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:45 smithi151 ceph-mon[32923]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:47.625 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:47 smithi094 ceph-mon[29291]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:47.625 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:47 smithi094 ceph-mon[40484]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:47 smithi151 ceph-mon[32923]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:49.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:49 smithi094 ceph-mon[29291]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:49 smithi094 ceph-mon[40484]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:49 smithi151 ceph-mon[32923]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:51.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:51 smithi094 ceph-mon[29291]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:51 smithi094 ceph-mon[40484]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:51 smithi151 ceph-mon[32923]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:51.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:51.928+0000 7f59717fa700 1 -- 172.21.15.94:0/1741235534 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 4116407425) 0x7f5960000bc0 con 0x7f595c06eb70 2024-03-12T03:10:51.931 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 1 on host 'smithi094' 2024-03-12T03:10:51.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:51.931+0000 7f597a561700 1 -- 172.21.15.94:0/1741235534 >> v1:172.21.15.94:6800/1299282424 conn(0x7f595c06eb70 legacy=0x7f595c071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:51.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:51.931+0000 7f597a561700 1 -- 172.21.15.94:0/1741235534 >> v1:172.21.15.94:6790/0 conn(0x7f5974075d90 legacy=0x7f597419f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:51.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:51.931+0000 7f597a561700 1 -- 172.21.15.94:0/1741235534 shutdown_connections 2024-03-12T03:10:51.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:51.931+0000 7f597a561700 1 -- 172.21.15.94:0/1741235534 >> 172.21.15.94:0/1741235534 conn(0x7f59740fd190 msgr2=0x7f59740fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:10:51.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:51.931+0000 7f597a561700 1 -- 172.21.15.94:0/1741235534 shutdown_connections 2024-03-12T03:10:51.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:51.931+0000 7f597a561700 1 -- 172.21.15.94:0/1741235534 wait complete. 2024-03-12T03:10:52.531 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:52 smithi094 ceph-mon[29291]: from='osd.1 v1:172.21.15.94:6805/1056655004' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:10:52.531 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:52 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:52.531 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:52 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:52.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:52 smithi094 ceph-mon[40484]: from='osd.1 v1:172.21.15.94:6805/1056655004' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:10:52.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:52 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:52.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:52 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:52.558 DEBUG:teuthology.orchestra.run.smithi094:osd.1> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.1.service 2024-03-12T03:10:52.561 INFO:tasks.cephadm:Deploying osd.2 on smithi094 with /dev/vg_nvme/lv_2... 2024-03-12T03:10:52.561 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T03:10:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:52 smithi151 ceph-mon[32923]: from='osd.1 v1:172.21.15.94:6805/1056655004' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:10:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:52 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:52 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:52.854 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:10:53.638 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[29291]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:53.639 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[29291]: from='osd.1 v1:172.21.15.94:6805/1056655004' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:10:53.639 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[29291]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:10:53.639 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[29291]: from='osd.1 v1:172.21.15.94:6805/1056655004' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:10:53.639 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:53.639 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:53.639 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:53.639 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[40484]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:53.639 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[40484]: from='osd.1 v1:172.21.15.94:6805/1056655004' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:10:53.640 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[40484]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:10:53.640 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[40484]: from='osd.1 v1:172.21.15.94:6805/1056655004' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:10:53.640 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:53.640 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:53.640 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:53.640 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1[50221]: 2024-03-12T03:10:53.447+0000 7f2b440f7700 -1 osd.1 0 waiting for initial osdmap 2024-03-12T03:10:53.640 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:10:53 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1[50221]: 2024-03-12T03:10:53.449+0000 7f2b3feea700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:10:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:53 smithi151 ceph-mon[32923]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:53 smithi151 ceph-mon[32923]: from='osd.1 v1:172.21.15.94:6805/1056655004' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:10:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:53 smithi151 ceph-mon[32923]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:10:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:53 smithi151 ceph-mon[32923]: from='osd.1 v1:172.21.15.94:6805/1056655004' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:10:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:54.452 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:54 smithi094 ceph-mon[29291]: from='osd.1 v1:172.21.15.94:6805/1056655004' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-03-12T03:10:54.453 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:54 smithi094 ceph-mon[29291]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:10:54.453 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:54 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:54.453 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:54 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:54.453 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:54 smithi094 ceph-mon[40484]: from='osd.1 v1:172.21.15.94:6805/1056655004' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-03-12T03:10:54.453 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:54 smithi094 ceph-mon[40484]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:10:54.453 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:54 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:54.453 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:54 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:54 smithi151 ceph-mon[32923]: from='osd.1 v1:172.21.15.94:6805/1056655004' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-03-12T03:10:54.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:54 smithi151 ceph-mon[32923]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:10:54.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:54 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:54.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:54 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:54.772 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T03:10:54.773 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T03:10:54.773 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-03-12T03:10:54.773 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-03-12T03:10:54.773 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126838 s, 827 MB/s 2024-03-12T03:10:54.773 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-03-12T03:10:55.437 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi094:vg_nvme/lv_2 2024-03-12T03:10:55.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[29291]: purged_snaps scrub starts 2024-03-12T03:10:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[29291]: purged_snaps scrub ok 2024-03-12T03:10:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[29291]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[29291]: osd.1 v1:172.21.15.94:6805/1056655004 boot 2024-03-12T03:10:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[29291]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T03:10:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[40484]: purged_snaps scrub starts 2024-03-12T03:10:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[40484]: purged_snaps scrub ok 2024-03-12T03:10:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[40484]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:55.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:55.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[40484]: osd.1 v1:172.21.15.94:6805/1056655004 boot 2024-03-12T03:10:55.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[40484]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T03:10:55.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:55 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:55 smithi151 ceph-mon[32923]: purged_snaps scrub starts 2024-03-12T03:10:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:55 smithi151 ceph-mon[32923]: purged_snaps scrub ok 2024-03-12T03:10:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:55 smithi151 ceph-mon[32923]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:10:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:55 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:55 smithi151 ceph-mon[32923]: osd.1 v1:172.21.15.94:6805/1056655004 boot 2024-03-12T03:10:55.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:55 smithi151 ceph-mon[32923]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T03:10:55.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:55 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:10:56.214 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:10:56.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:56 smithi094 ceph-mon[29291]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:10:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:56 smithi094 ceph-mon[40484]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:10:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:56 smithi151 ceph-mon[32923]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:10:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:57 smithi151 ceph-mon[32923]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:10:57.921 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:57 smithi094 ceph-mon[40484]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:10:57.922 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:57 smithi094 ceph-mon[29291]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:10:58.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.951+0000 7f855f710700 1 -- 172.21.15.94:0/2533147434 >> v1:172.21.15.94:6790/0 conn(0x7f8558106020 legacy=0x7f8558108470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:58.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.951+0000 7f855f710700 1 -- 172.21.15.94:0/2533147434 shutdown_connections 2024-03-12T03:10:58.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.951+0000 7f855f710700 1 -- 172.21.15.94:0/2533147434 >> 172.21.15.94:0/2533147434 conn(0x7f85580fd190 msgr2=0x7f85580ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:10:58.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.951+0000 7f855f710700 1 -- 172.21.15.94:0/2533147434 shutdown_connections 2024-03-12T03:10:58.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.951+0000 7f855f710700 1 -- 172.21.15.94:0/2533147434 wait complete. 2024-03-12T03:10:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.952+0000 7f855f710700 1 Processor -- start 2024-03-12T03:10:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.952+0000 7f855f710700 1 -- start start 2024-03-12T03:10:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.952+0000 7f855f710700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8558197b80 con 0x7f8558106020 2024-03-12T03:10:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.952+0000 7f855f710700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f85581a3370 con 0x7f8558109d60 2024-03-12T03:10:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.952+0000 7f855f710700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f85581a4550 con 0x7f8558101bf0 2024-03-12T03:10:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.952+0000 7f855d4ac700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f8558101bf0 0x7f8558197030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:39580/0 (socket says 172.21.15.94:39580) 2024-03-12T03:10:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.952+0000 7f855d4ac700 1 -- 172.21.15.94:0/3652895113 learned_addr learned my addr 172.21.15.94:0/3652895113 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:10:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.952+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689132242 0 0) 0x7f8558197b80 con 0x7f8558106020 2024-03-12T03:10:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.952+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85440035b0 con 0x7f8558106020 2024-03-12T03:10:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211934640 0 0) 0x7f85581a4550 con 0x7f8558101bf0 2024-03-12T03:10:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8558197b80 con 0x7f8558101bf0 2024-03-12T03:10:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152574993 0 0) 0x7f85581a3370 con 0x7f8558109d60 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85581a4550 con 0x7f8558109d60 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 375088333 0 0) 0x7f8558197b80 con 0x7f8558101bf0 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85581a3370 con 0x7f8558101bf0 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8554001770 con 0x7f8558101bf0 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1085621431 0 0) 0x7f85440035b0 con 0x7f8558106020 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8558197b80 con 0x7f8558106020 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8548004430 con 0x7f8558106020 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1772181679 0 0) 0x7f85581a3370 con 0x7f8558101bf0 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 >> v1:172.21.15.151:6789/0 conn(0x7f8558109d60 legacy=0x7f85581a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 >> v1:172.21.15.94:6789/0 conn(0x7f8558106020 legacy=0x7f855819d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85581a5770 con 0x7f8558101bf0 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f855f710700 1 -- 172.21.15.94:0/3652895113 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85581a4780 con 0x7f8558101bf0 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f855f710700 1 -- 172.21.15.94:0/3652895113 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85581a4cc0 con 0x7f8558101bf0 2024-03-12T03:10:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f8554003b30 con 0x7f8558101bf0 2024-03-12T03:10:58.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.953+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8554005bc0 con 0x7f8558101bf0 2024-03-12T03:10:58.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.954+0000 7f855f710700 1 -- 172.21.15.94:0/3652895113 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853c005150 con 0x7f8558101bf0 2024-03-12T03:10:58.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.954+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 14) v1 ==== 91663+0+0 (unknown 3714127607 0 0) 0x7f85540036a0 con 0x7f8558101bf0 2024-03-12T03:10:58.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.955+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(14..14 src has 1..14) v4 ==== 1833+0+0 (unknown 2998257193 0 0) 0x7f8554087db0 con 0x7f8558101bf0 2024-03-12T03:10:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:58.960+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f8554056960 con 0x7f8558101bf0 2024-03-12T03:10:59.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:10:59.166+0000 7f855f710700 1 -- 172.21.15.94:0/3652895113 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f853c000bc0 con 0x7f854406eac0 2024-03-12T03:10:59.170 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: Detected new or changed devices on smithi094 2024-03-12T03:10:59.170 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.170 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.170 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:10:59.170 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:10:59.170 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: Adjusting osd_memory_target on smithi094 to 8032M 2024-03-12T03:10:59.170 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.170 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:59.170 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:59.171 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.171 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:59.171 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:59.171 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:59.171 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.171 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[29291]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:10:59.171 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: Detected new or changed devices on smithi094 2024-03-12T03:10:59.171 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.171 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.171 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:10:59.171 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:10:59.171 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: Adjusting osd_memory_target on smithi094 to 8032M 2024-03-12T03:10:59.172 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.172 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:59.172 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:59.172 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.172 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:59.172 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:59.172 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:59.172 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.172 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:10:58 smithi094 ceph-mon[40484]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:10:59.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: Detected new or changed devices on smithi094 2024-03-12T03:10:59.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:10:59.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:10:59.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: Adjusting osd_memory_target on smithi094 to 8032M 2024-03-12T03:10:59.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:59.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:59.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:10:59.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:10:59.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:10:59.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:10:59.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:10:58 smithi151 ceph-mon[32923]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:00 smithi094 ceph-mon[29291]: from='client.34166 v1:172.21.15.94:0/3652895113' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:11:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:00 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:11:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:00 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:11:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:00 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:00.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:00 smithi094 ceph-mon[40484]: from='client.34166 v1:172.21.15.94:0/3652895113' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:11:00.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:00 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:11:00.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:00 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:11:00.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:00 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:00.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:00 smithi151 ceph-mon[32923]: from='client.34166 v1:172.21.15.94:0/3652895113' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:11:00.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:00 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:11:00.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:00 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:11:00.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:00 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:01.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:01 smithi094 ceph-mon[40484]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:01.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:01 smithi094 ceph-mon[29291]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:01.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:01 smithi151 ceph-mon[32923]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:02.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3417541673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b0ab939-b3de-4821-9c65-348f5172981d"}]: dispatch 2024-03-12T03:11:02.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3417541673' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b0ab939-b3de-4821-9c65-348f5172981d"}]': finished 2024-03-12T03:11:02.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:02 smithi094 ceph-mon[29291]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T03:11:02.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3417541673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b0ab939-b3de-4821-9c65-348f5172981d"}]: dispatch 2024-03-12T03:11:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3417541673' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b0ab939-b3de-4821-9c65-348f5172981d"}]': finished 2024-03-12T03:11:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:02 smithi094 ceph-mon[40484]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T03:11:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:02.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3417541673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b0ab939-b3de-4821-9c65-348f5172981d"}]: dispatch 2024-03-12T03:11:02.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3417541673' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b0ab939-b3de-4821-9c65-348f5172981d"}]': finished 2024-03-12T03:11:02.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:02 smithi151 ceph-mon[32923]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T03:11:02.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:03 smithi094 ceph-mon[29291]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3347585594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:11:03.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:03 smithi094 ceph-mon[40484]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:03.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3347585594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:11:03.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:03 smithi151 ceph-mon[32923]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:03.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3347585594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:11:05.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:05 smithi094 ceph-mon[29291]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:05.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:05 smithi094 ceph-mon[40484]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:05 smithi151 ceph-mon[32923]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:07 smithi094 ceph-mon[29291]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:07.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:07 smithi094 ceph-mon[40484]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:07.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:07 smithi151 ceph-mon[32923]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:09 smithi094 ceph-mon[29291]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:09 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:11:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:09 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:09 smithi094 ceph-mon[29291]: Deploying daemon osd.2 on smithi094 2024-03-12T03:11:09.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:09 smithi094 ceph-mon[40484]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:09.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:09 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:11:09.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:09 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:09.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:09 smithi094 ceph-mon[40484]: Deploying daemon osd.2 on smithi094 2024-03-12T03:11:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:09 smithi151 ceph-mon[32923]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:09 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:11:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:09 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:09 smithi151 ceph-mon[32923]: Deploying daemon osd.2 on smithi094 2024-03-12T03:11:11.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:11 smithi094 ceph-mon[40484]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:11 smithi094 ceph-mon[29291]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:11.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:11 smithi151 ceph-mon[32923]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:13.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:13 smithi094 ceph-mon[29291]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:13.508 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:13 smithi094 ceph-mon[40484]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:13.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:13 smithi151 ceph-mon[32923]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:15.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:15 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:15.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:15 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:15.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:15 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:11:15.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:15 smithi094 ceph-mon[29291]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:15.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:15 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:15.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:15 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:15.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:15 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:11:15.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:15 smithi094 ceph-mon[40484]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:15.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:15 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:15.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:15 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:15.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:15 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:11:15.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:15 smithi151 ceph-mon[32923]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:17.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:17 smithi094 ceph-mon[29291]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:17 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:17 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:17 smithi094 ceph-mon[29291]: from='osd.2 v1:172.21.15.94:6809/2929979688' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:11:17.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:17 smithi094 ceph-mon[40484]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:17.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:17 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:17.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:17 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:17.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:17 smithi094 ceph-mon[40484]: from='osd.2 v1:172.21.15.94:6809/2929979688' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:11:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:17 smithi151 ceph-mon[32923]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:17 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:17.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:17 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:17.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:17 smithi151 ceph-mon[32923]: from='osd.2 v1:172.21.15.94:6809/2929979688' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:11:17.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:17.874+0000 7f854e7fc700 1 -- 172.21.15.94:0/3652895113 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 291854173) 0x7f853c000bc0 con 0x7f854406eac0 2024-03-12T03:11:17.878 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 2 on host 'smithi094' 2024-03-12T03:11:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:17.877+0000 7f855f710700 1 -- 172.21.15.94:0/3652895113 >> v1:172.21.15.94:6800/1299282424 conn(0x7f854406eac0 legacy=0x7f8544070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:11:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:17.877+0000 7f855f710700 1 -- 172.21.15.94:0/3652895113 >> v1:172.21.15.94:6790/0 conn(0x7f8558101bf0 legacy=0x7f8558197030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:11:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:17.877+0000 7f855f710700 1 -- 172.21.15.94:0/3652895113 shutdown_connections 2024-03-12T03:11:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:17.877+0000 7f855f710700 1 -- 172.21.15.94:0/3652895113 >> 172.21.15.94:0/3652895113 conn(0x7f85580fd190 msgr2=0x7f8558108ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:11:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:17.877+0000 7f855f710700 1 -- 172.21.15.94:0/3652895113 shutdown_connections 2024-03-12T03:11:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:17.877+0000 7f855f710700 1 -- 172.21.15.94:0/3652895113 wait complete. 2024-03-12T03:11:18.436 DEBUG:teuthology.orchestra.run.smithi094:osd.2> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.2.service 2024-03-12T03:11:18.439 INFO:tasks.cephadm:Deploying osd.3 on smithi094 with /dev/vg_nvme/lv_1... 2024-03-12T03:11:18.439 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-12T03:11:18.701 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:18.701 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:11:18.701 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:18.701 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:18.701 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:11:18.701 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:18.701 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[29291]: from='osd.2 v1:172.21.15.94:6809/2929979688' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:11:18.701 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[29291]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:11:18.702 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[29291]: from='osd.2 v1:172.21.15.94:6809/2929979688' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:11:18.702 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:18.702 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:18.702 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:11:18.702 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:18.702 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:18.702 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:11:18.702 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:18.702 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[40484]: from='osd.2 v1:172.21.15.94:6809/2929979688' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:11:18.702 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[40484]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:11:18.702 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[40484]: from='osd.2 v1:172.21.15.94:6809/2929979688' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:11:18.702 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:18 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:18 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:18 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:11:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:18 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:18 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:18 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:11:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:18 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:18 smithi151 ceph-mon[32923]: from='osd.2 v1:172.21.15.94:6809/2929979688' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:11:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:18 smithi151 ceph-mon[32923]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:11:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:18 smithi151 ceph-mon[32923]: from='osd.2 v1:172.21.15.94:6809/2929979688' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:11:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:18 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:19.407 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:11:19 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2[56245]: 2024-03-12T03:11:19.090+0000 7fcfc57d9700 -1 osd.2 0 waiting for initial osdmap 2024-03-12T03:11:19.407 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:11:19 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2[56245]: 2024-03-12T03:11:19.093+0000 7fcfc15cc700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:11:19.615 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:11:19.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:19 smithi094 ceph-mon[40484]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:19 smithi094 ceph-mon[40484]: from='osd.2 v1:172.21.15.94:6809/2929979688' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-03-12T03:11:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:19 smithi094 ceph-mon[40484]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:11:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:19 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:19 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:19 smithi094 ceph-mon[29291]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:19 smithi094 ceph-mon[29291]: from='osd.2 v1:172.21.15.94:6809/2929979688' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-03-12T03:11:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:19 smithi094 ceph-mon[29291]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:11:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:19 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:19 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:19 smithi151 ceph-mon[32923]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:19 smithi151 ceph-mon[32923]: from='osd.2 v1:172.21.15.94:6809/2929979688' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-03-12T03:11:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:19 smithi151 ceph-mon[32923]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:11:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:19 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:19 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:20.918 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:20 smithi094 ceph-mon[29291]: purged_snaps scrub starts 2024-03-12T03:11:20.918 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:20 smithi094 ceph-mon[29291]: purged_snaps scrub ok 2024-03-12T03:11:20.918 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:20 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:20.918 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:20 smithi094 ceph-mon[29291]: osd.2 v1:172.21.15.94:6809/2929979688 boot 2024-03-12T03:11:20.918 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:20 smithi094 ceph-mon[29291]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T03:11:20.918 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:20 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:20.919 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:20 smithi094 ceph-mon[40484]: purged_snaps scrub starts 2024-03-12T03:11:20.919 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:20 smithi094 ceph-mon[40484]: purged_snaps scrub ok 2024-03-12T03:11:20.919 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:20 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:20.919 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:20 smithi094 ceph-mon[40484]: osd.2 v1:172.21.15.94:6809/2929979688 boot 2024-03-12T03:11:20.919 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:20 smithi094 ceph-mon[40484]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T03:11:20.919 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:20 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:20 smithi151 ceph-mon[32923]: purged_snaps scrub starts 2024-03-12T03:11:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:20 smithi151 ceph-mon[32923]: purged_snaps scrub ok 2024-03-12T03:11:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:20 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:20 smithi151 ceph-mon[32923]: osd.2 v1:172.21.15.94:6809/2929979688 boot 2024-03-12T03:11:20.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:20 smithi151 ceph-mon[32923]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T03:11:20.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:20 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:11:21.795 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:21 smithi094 ceph-mon[29291]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:21.795 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:21 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:11:21.796 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:21 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T03:11:21.796 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:21 smithi094 ceph-mon[29291]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:11:21.796 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:21 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:11:21.796 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:21 smithi094 ceph-mon[40484]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:21.796 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:21 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:11:21.797 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:21 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T03:11:21.797 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:21 smithi094 ceph-mon[40484]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:11:21.797 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:21 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:11:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:21 smithi151 ceph-mon[32923]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:11:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:21 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:11:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:21 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T03:11:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:21 smithi151 ceph-mon[32923]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:11:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:21 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:11:22.650 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-12T03:11:22.650 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-12T03:11:22.650 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-03-12T03:11:22.650 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-03-12T03:11:22.650 INFO:teuthology.orchestra.run.smithi094.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00833533 s, 1.3 GB/s 2024-03-12T03:11:22.650 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-03-12T03:11:22.931 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:11:22 smithi094 sudo[60415]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:11:22.931 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:11:22 smithi094 sudo[60415]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:11:22.931 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:11:22 smithi094 sudo[60415]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:11:22.931 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:11:22 smithi094 sudo[60415]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:11:22.932 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: Detected new or changed devices on smithi094 2024-03-12T03:11:22.932 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:22.932 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:22.932 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:22.932 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:22.932 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:22.932 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: Adjusting osd_memory_target on smithi094 to 5354M 2024-03-12T03:11:22.932 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:22.932 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:22.932 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:11:22.932 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:22.933 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:11:22.933 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:11:22.933 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[40484]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:22.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: Detected new or changed devices on smithi094 2024-03-12T03:11:22.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:22.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:22.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:22.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:22.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:22.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: Adjusting osd_memory_target on smithi094 to 5354M 2024-03-12T03:11:22.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:22.933 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:22.934 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:11:22.934 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:22.934 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:11:22.934 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:11:22.934 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:22 smithi094 ceph-mon[29291]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:23.198 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:11:22 smithi094 sudo[60433]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:11:23.199 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:11:22 smithi094 sudo[60433]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:11:23.199 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:11:22 smithi094 sudo[60433]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:11:23.199 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:11:22 smithi094 sudo[60433]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:11:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: Detected new or changed devices on smithi094 2024-03-12T03:11:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: Adjusting osd_memory_target on smithi094 to 5354M 2024-03-12T03:11:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:11:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:11:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:11:23.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:22 smithi151 ceph-mon[32923]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:23.319 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi094:vg_nvme/lv_1 2024-03-12T03:11:23.460 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60436]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:11:23.460 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60436]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:11:23.460 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60436]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:11:23.460 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60436]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:11:23.565 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:11:23.717 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60457]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:11:23.717 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60457]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:11:23.717 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60457]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:11:23.717 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60457]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:11:23.979 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:23 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:23.979 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:23 smithi094 ceph-mon[29291]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:11:23.980 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:23 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:23.980 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:23 smithi094 ceph-mon[40484]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:11:23.980 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60517]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:11:23.980 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60517]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:11:23.980 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60517]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:11:23.980 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60517]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:11:24.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:23 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:24.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:23 smithi151 ceph-mon[32923]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:11:24.242 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:24 smithi094 sudo[60541]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:11:24.242 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:24 smithi094 sudo[60541]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:11:24.242 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:24 smithi094 sudo[60541]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:11:24.242 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60538]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:11:24.243 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60538]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:11:24.243 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60538]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:11:24.243 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:11:23 smithi094 sudo[60538]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:11:24.626 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:24 smithi094 sudo[60541]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:11:24.626 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:24 smithi094 sudo[60546]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:11:24.626 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:24 smithi094 sudo[60546]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:11:24.626 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:24 smithi094 sudo[60546]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:11:24.626 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:24 smithi094 sudo[60546]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:11:24.993 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:24 smithi151 sudo[35471]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:11:24.993 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:24 smithi151 sudo[35471]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:11:24.993 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:24 smithi151 sudo[35471]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:11:24.993 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:24 smithi151 sudo[35471]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:11:25.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.130+0000 7f77ef6f5700 1 -- 172.21.15.94:0/1596484915 >> v1:172.21.15.94:6789/0 conn(0x7f77e8108c40 legacy=0x7f77e810b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:11:25.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.130+0000 7f77ef6f5700 1 -- 172.21.15.94:0/1596484915 shutdown_connections 2024-03-12T03:11:25.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.130+0000 7f77ef6f5700 1 -- 172.21.15.94:0/1596484915 >> 172.21.15.94:0/1596484915 conn(0x7f77e80fc070 msgr2=0x7f77e80fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:11:25.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.131+0000 7f77ef6f5700 1 -- 172.21.15.94:0/1596484915 shutdown_connections 2024-03-12T03:11:25.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.131+0000 7f77ef6f5700 1 -- 172.21.15.94:0/1596484915 wait complete. 2024-03-12T03:11:25.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.131+0000 7f77ef6f5700 1 Processor -- start 2024-03-12T03:11:25.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.131+0000 7f77ef6f5700 1 -- start start 2024-03-12T03:11:25.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.131+0000 7f77ef6f5700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f77e806a640 con 0x7f77e8108c40 2024-03-12T03:11:25.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.131+0000 7f77ef6f5700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f77e81a34a0 con 0x7f77e8100ad0 2024-03-12T03:11:25.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.131+0000 7f77ef6f5700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f77e81a46e0 con 0x7f77e8104f00 2024-03-12T03:11:25.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.131+0000 7f77edc92700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f77e8108c40 0x7f77e81a0d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58826/0 (socket says 172.21.15.94:58826) 2024-03-12T03:11:25.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.131+0000 7f77edc92700 1 -- 172.21.15.94:0/2719630616 learned_addr learned my addr 172.21.15.94:0/2719630616 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:11:25.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343921112 0 0) 0x7f77e81a34a0 con 0x7f77e8100ad0 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d40035b0 con 0x7f77e8100ad0 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922647019 0 0) 0x7f77e806a640 con 0x7f77e8108c40 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77e81a34a0 con 0x7f77e8108c40 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1642569063 0 0) 0x7f77d40035b0 con 0x7f77e8100ad0 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77e806a640 con 0x7f77e8100ad0 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f77e4001720 con 0x7f77e8100ad0 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1415168178 0 0) 0x7f77e81a34a0 con 0x7f77e8108c40 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77d40035b0 con 0x7f77e8108c40 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f77e00017b0 con 0x7f77e8108c40 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 370617911 0 0) 0x7f77e806a640 con 0x7f77e8100ad0 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 >> v1:172.21.15.94:6790/0 conn(0x7f77e8104f00 legacy=0x7f77e819d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 >> v1:172.21.15.94:6789/0 conn(0x7f77e8108c40 legacy=0x7f77e81a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e81a5920 con 0x7f77e8100ad0 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77ef6f5700 1 -- 172.21.15.94:0/2719630616 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e81a36d0 con 0x7f77e8100ad0 2024-03-12T03:11:25.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77ef6f5700 1 -- 172.21.15.94:0/2719630616 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e81a3c10 con 0x7f77e8100ad0 2024-03-12T03:11:25.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f77e4001d10 con 0x7f77e8100ad0 2024-03-12T03:11:25.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.132+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f77e4004910 con 0x7f77e8100ad0 2024-03-12T03:11:25.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.133+0000 7f77ef6f5700 1 -- 172.21.15.94:0/2719630616 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77cc005150 con 0x7f77e8100ad0 2024-03-12T03:11:25.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.134+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 15) v1 ==== 91719+0+0 (unknown 3315827787 0 0) 0x7f77e401b1a0 con 0x7f77e8100ad0 2024-03-12T03:11:25.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.134+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(21..21 src has 1..21) v4 ==== 2511+0+0 (unknown 1351054305 0 0) 0x7f77e4086fe0 con 0x7f77e8100ad0 2024-03-12T03:11:25.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.139+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f77e40556d0 con 0x7f77e8100ad0 2024-03-12T03:11:25.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:11:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:11:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:11:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[40484]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:11:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[40484]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[40484]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:11:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[40484]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-12T03:11:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:11:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:11:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:11:25.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:11:25.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:11:25.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:11:25.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[29291]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:11:25.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[29291]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:25.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[29291]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:11:25.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[29291]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-12T03:11:25.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:11:25.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:11:25.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:25 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:11:25.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:25.351+0000 7f77ef6f5700 1 -- 172.21.15.94:0/2719630616 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f77cc000bc0 con 0x7f77d406eb50 2024-03-12T03:11:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:24 smithi151 sudo[35474]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:11:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:24 smithi151 sudo[35474]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:11:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:24 smithi151 sudo[35474]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:11:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:24 smithi151 sudo[35474]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:11:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:25 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:11:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:25 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:11:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:25 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:11:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:25 smithi151 ceph-mon[32923]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:11:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:25 smithi151 ceph-mon[32923]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:25 smithi151 ceph-mon[32923]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:11:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:25 smithi151 ceph-mon[32923]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-12T03:11:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:25 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:11:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:25 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:11:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:25 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:11:25.528 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 sudo[60596]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:11:25.528 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 sudo[60596]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:11:25.528 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 sudo[60596]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:11:25.529 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 sudo[60596]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:11:25.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 sudo[60666]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:11:25.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 sudo[60666]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:11:25.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 sudo[60666]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:11:25.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:25 smithi094 sudo[60666]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:11:26.291 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[29291]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:11:26.291 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[29291]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:11:26.291 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:26.291 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:11:26.291 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:11:26.291 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[29291]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:11:26.291 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:11:26.291 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[29291]: from='client.24205 v1:172.21.15.94:0/2719630616' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:11:26.291 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:11:26.291 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:11:26.291 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:26.292 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[29291]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:11:26.292 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[40484]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:11:26.292 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[40484]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:11:26.292 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:26.292 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:11:26.292 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:11:26.292 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[40484]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:11:26.292 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:11:26.292 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[40484]: from='client.24205 v1:172.21.15.94:0/2719630616' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:11:26.292 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:11:26.292 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:11:26.293 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:26.293 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:26 smithi094 ceph-mon[40484]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:11:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:26 smithi151 ceph-mon[32923]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:11:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:26 smithi151 ceph-mon[32923]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:11:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:26 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:26 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:11:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:26 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:11:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:26 smithi151 ceph-mon[32923]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:11:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:26 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:11:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:26 smithi151 ceph-mon[32923]: from='client.24205 v1:172.21.15.94:0/2719630616' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:11:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:26 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:11:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:26 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:11:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:26 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:26 smithi151 ceph-mon[32923]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:11:27.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:27 smithi094 ceph-mon[29291]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:27.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:27 smithi094 ceph-mon[40484]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:27.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:27 smithi151 ceph-mon[32923]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:28.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3772346265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3fd5c9c8-6304-4233-a4df-8c4c4bda5f79"}]: dispatch 2024-03-12T03:11:28.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3772346265' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3fd5c9c8-6304-4233-a4df-8c4c4bda5f79"}]': finished 2024-03-12T03:11:28.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:28 smithi094 ceph-mon[40484]: osdmap e22: 4 total, 3 up, 4 in 2024-03-12T03:11:28.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:28 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3772346265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3fd5c9c8-6304-4233-a4df-8c4c4bda5f79"}]: dispatch 2024-03-12T03:11:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3772346265' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3fd5c9c8-6304-4233-a4df-8c4c4bda5f79"}]': finished 2024-03-12T03:11:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:28 smithi094 ceph-mon[29291]: osdmap e22: 4 total, 3 up, 4 in 2024-03-12T03:11:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:28 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:28.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3772346265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3fd5c9c8-6304-4233-a4df-8c4c4bda5f79"}]: dispatch 2024-03-12T03:11:28.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3772346265' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3fd5c9c8-6304-4233-a4df-8c4c4bda5f79"}]': finished 2024-03-12T03:11:28.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:28 smithi151 ceph-mon[32923]: osdmap e22: 4 total, 3 up, 4 in 2024-03-12T03:11:28.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:28 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:29.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:29 smithi094 ceph-mon[29291]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:29.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3524257620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:11:29.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:29 smithi094 ceph-mon[40484]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:29.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3524257620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:11:29.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:29 smithi151 ceph-mon[32923]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:29.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3524257620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:11:31.610 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:31 smithi094 ceph-mon[29291]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:31.611 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:31 smithi094 ceph-mon[40484]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:31.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:31 smithi151 ceph-mon[32923]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:33.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:33 smithi094 ceph-mon[29291]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-12T03:11:33.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:33 smithi094 ceph-mon[40484]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-12T03:11:33.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:33 smithi151 ceph-mon[32923]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-12T03:11:35.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:35 smithi094 ceph-mon[29291]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-12T03:11:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:35 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T03:11:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:35 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:35.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:35 smithi094 ceph-mon[40484]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-12T03:11:35.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:35 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T03:11:35.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:35 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:35 smithi151 ceph-mon[32923]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-12T03:11:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:35 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T03:11:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:35 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:36.675 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:36 smithi094 ceph-mon[29291]: Deploying daemon osd.3 on smithi094 2024-03-12T03:11:36.675 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:36 smithi094 ceph-mon[40484]: Deploying daemon osd.3 on smithi094 2024-03-12T03:11:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:36 smithi151 ceph-mon[32923]: Deploying daemon osd.3 on smithi094 2024-03-12T03:11:37.669 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:37 smithi094 ceph-mon[29291]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-12T03:11:37.670 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:37 smithi094 ceph-mon[40484]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-12T03:11:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:37 smithi151 ceph-mon[32923]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-12T03:11:39.554 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:39 smithi094 ceph-mon[29291]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 895 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-12T03:11:39.554 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:39 smithi094 ceph-mon[40484]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 895 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-12T03:11:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:39 smithi151 ceph-mon[32923]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 895 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-12T03:11:41.423 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:41 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:41.423 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:41 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:11:41.423 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:41 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:41.423 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:41 smithi094 ceph-mon[29291]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-12T03:11:41.424 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:41 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:41.424 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:41 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:11:41.424 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:41 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:41.424 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:41 smithi094 ceph-mon[40484]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-12T03:11:41.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:41 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:41.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:41 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:11:41.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:41 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:41.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:41 smithi151 ceph-mon[32923]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-12T03:11:43.698 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:43 smithi094 ceph-mon[40484]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-12T03:11:43.699 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:43 smithi094 ceph-mon[40484]: from='osd.3 v1:172.21.15.94:6813/4235334318' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:11:43.699 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:43 smithi094 ceph-mon[40484]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:11:43.699 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:43 smithi094 ceph-mon[29291]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-12T03:11:43.699 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:43 smithi094 ceph-mon[29291]: from='osd.3 v1:172.21.15.94:6813/4235334318' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:11:43.699 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:43 smithi094 ceph-mon[29291]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:11:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:43 smithi151 ceph-mon[32923]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-12T03:11:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:43 smithi151 ceph-mon[32923]: from='osd.3 v1:172.21.15.94:6813/4235334318' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:11:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:43 smithi151 ceph-mon[32923]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:11:43.758 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 3 on host 'smithi094' 2024-03-12T03:11:43.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:43.755+0000 7f77de7fc700 1 -- 172.21.15.94:0/2719630616 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 1299233257) 0x7f77cc000bc0 con 0x7f77d406eb50 2024-03-12T03:11:43.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:43.757+0000 7f77ef6f5700 1 -- 172.21.15.94:0/2719630616 >> v1:172.21.15.94:6800/1299282424 conn(0x7f77d406eb50 legacy=0x7f77d4071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:11:43.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:43.757+0000 7f77ef6f5700 1 -- 172.21.15.94:0/2719630616 >> v1:172.21.15.151:6789/0 conn(0x7f77e8100ad0 legacy=0x7f77e8199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:11:43.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:43.757+0000 7f77ef6f5700 1 -- 172.21.15.94:0/2719630616 shutdown_connections 2024-03-12T03:11:43.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:43.757+0000 7f77ef6f5700 1 -- 172.21.15.94:0/2719630616 >> 172.21.15.94:0/2719630616 conn(0x7f77e80fc070 msgr2=0x7f77e810b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:11:43.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:43.757+0000 7f77ef6f5700 1 -- 172.21.15.94:0/2719630616 shutdown_connections 2024-03-12T03:11:43.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:11:43.757+0000 7f77ef6f5700 1 -- 172.21.15.94:0/2719630616 wait complete. 2024-03-12T03:11:44.442 DEBUG:teuthology.orchestra.run.smithi094:osd.3> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.3.service 2024-03-12T03:11:44.444 INFO:tasks.cephadm:Deploying osd.4 on smithi151 with /dev/vg_nvme/lv_4... 2024-03-12T03:11:44.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T03:11:44.689 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:11:44.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[40484]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:11:44.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[40484]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T03:11:44.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[40484]: from='osd.3 v1:172.21.15.94:6813/4235334318' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:11:44.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[40484]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:11:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[40484]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-03-12T03:11:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[40484]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T03:11:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[29291]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:11:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[29291]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T03:11:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[29291]: from='osd.3 v1:172.21.15.94:6813/4235334318' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:11:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[29291]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:11:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[29291]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-03-12T03:11:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[29291]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T03:11:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:44 smithi151 ceph-mon[32923]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:11:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:44 smithi151 ceph-mon[32923]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T03:11:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:44 smithi151 ceph-mon[32923]: from='osd.3 v1:172.21.15.94:6813/4235334318' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:11:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:44 smithi151 ceph-mon[32923]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-03-12T03:11:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:44.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:44.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:44 smithi151 ceph-mon[32923]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-03-12T03:11:44.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:44 smithi151 ceph-mon[32923]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T03:11:44.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:45.612 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:45 smithi094 ceph-mon[40484]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:45.612 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:45 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:45.612 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:45 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:45.612 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:45 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:45.613 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:45 smithi094 ceph-mon[29291]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:45.613 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:45 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:45.613 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:45 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:45.613 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:45 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:45.693 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:45 smithi151 ceph-mon[32923]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:45.693 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:45 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:45.693 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:45 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:45.693 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:45 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:46.554 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T03:11:46.554 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T03:11:46.554 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-03-12T03:11:46.554 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-03-12T03:11:46.554 INFO:teuthology.orchestra.run.smithi151.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114132 s, 919 MB/s 2024-03-12T03:11:46.554 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-03-12T03:11:46.641 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:46 smithi094 ceph-mon[40484]: purged_snaps scrub starts 2024-03-12T03:11:46.641 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:46 smithi094 ceph-mon[40484]: purged_snaps scrub ok 2024-03-12T03:11:46.641 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:46 smithi094 ceph-mon[40484]: osd.3 v1:172.21.15.94:6813/4235334318 boot 2024-03-12T03:11:46.641 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:46 smithi094 ceph-mon[40484]: osdmap e25: 4 total, 4 up, 4 in 2024-03-12T03:11:46.641 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:46 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:46.641 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:46 smithi094 ceph-mon[29291]: purged_snaps scrub starts 2024-03-12T03:11:46.641 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:46 smithi094 ceph-mon[29291]: purged_snaps scrub ok 2024-03-12T03:11:46.642 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:46 smithi094 ceph-mon[29291]: osd.3 v1:172.21.15.94:6813/4235334318 boot 2024-03-12T03:11:46.642 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:46 smithi094 ceph-mon[29291]: osdmap e25: 4 total, 4 up, 4 in 2024-03-12T03:11:46.642 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:46 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:46 smithi151 ceph-mon[32923]: purged_snaps scrub starts 2024-03-12T03:11:46.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:46 smithi151 ceph-mon[32923]: purged_snaps scrub ok 2024-03-12T03:11:46.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:46 smithi151 ceph-mon[32923]: osd.3 v1:172.21.15.94:6813/4235334318 boot 2024-03-12T03:11:46.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:46 smithi151 ceph-mon[32923]: osdmap e25: 4 total, 4 up, 4 in 2024-03-12T03:11:46.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:46 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:11:47.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi151:vg_nvme/lv_4 2024-03-12T03:11:47.436 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:11:47.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:47 smithi094 ceph-mon[29291]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:47 smithi094 ceph-mon[29291]: osdmap e26: 4 total, 4 up, 4 in 2024-03-12T03:11:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:47 smithi094 ceph-mon[40484]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:47 smithi094 ceph-mon[40484]: osdmap e26: 4 total, 4 up, 4 in 2024-03-12T03:11:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:47 smithi151 ceph-mon[32923]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:11:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:47 smithi151 ceph-mon[32923]: osdmap e26: 4 total, 4 up, 4 in 2024-03-12T03:11:48.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.895+0000 7f9e513bd700 1 -- 172.21.15.151:0/4090504339 >> v1:172.21.15.151:6789/0 conn(0x7f9e4c108c20 legacy=0x7f9e4c10b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:11:48.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.895+0000 7f9e513bd700 1 -- 172.21.15.151:0/4090504339 shutdown_connections 2024-03-12T03:11:48.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.895+0000 7f9e513bd700 1 -- 172.21.15.151:0/4090504339 >> 172.21.15.151:0/4090504339 conn(0x7f9e4c0fb800 msgr2=0x7f9e4c0fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:11:48.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.895+0000 7f9e513bd700 1 -- 172.21.15.151:0/4090504339 shutdown_connections 2024-03-12T03:11:48.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.895+0000 7f9e513bd700 1 -- 172.21.15.151:0/4090504339 wait complete. 2024-03-12T03:11:48.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.895+0000 7f9e513bd700 1 Processor -- start 2024-03-12T03:11:48.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e513bd700 1 -- start start 2024-03-12T03:11:48.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e513bd700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e4c100a20 con 0x7f9e4c104f50 2024-03-12T03:11:48.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e513bd700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e4c1a3330 con 0x7f9e4c101000 2024-03-12T03:11:48.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e513bd700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e4c1a4550 con 0x7f9e4c108c20 2024-03-12T03:11:48.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e4affd700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f9e4c101000 0x7f9e4c199da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:41318/0 (socket says 172.21.15.151:41318) 2024-03-12T03:11:48.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e4affd700 1 -- 172.21.15.151:0/3219219254 learned_addr learned my addr 172.21.15.151:0/3219219254 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:11:48.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139723165 0 0) 0x7f9e4c1a3330 con 0x7f9e4c101000 2024-03-12T03:11:48.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e380035b0 con 0x7f9e4c101000 2024-03-12T03:11:48.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622884238 0 0) 0x7f9e4c100a20 con 0x7f9e4c104f50 2024-03-12T03:11:48.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e4c1a3330 con 0x7f9e4c104f50 2024-03-12T03:11:48.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235286648 0 0) 0x7f9e4c1a4550 con 0x7f9e4c108c20 2024-03-12T03:11:48.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e4c100a20 con 0x7f9e4c108c20 2024-03-12T03:11:48.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2964064252 0 0) 0x7f9e380035b0 con 0x7f9e4c101000 2024-03-12T03:11:48.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e4c1a4550 con 0x7f9e4c101000 2024-03-12T03:11:48.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f9e3c001870 con 0x7f9e4c101000 2024-03-12T03:11:48.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2416152390 0 0) 0x7f9e4c100a20 con 0x7f9e4c108c20 2024-03-12T03:11:48.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e380035b0 con 0x7f9e4c108c20 2024-03-12T03:11:48.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 710523319 0 0) 0x7f9e4c1a3330 con 0x7f9e4c104f50 2024-03-12T03:11:48.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.896+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e4c100a20 con 0x7f9e4c104f50 2024-03-12T03:11:48.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.897+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f9e44001950 con 0x7f9e4c108c20 2024-03-12T03:11:48.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.897+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 124050475 0 0) 0x7f9e4c1a4550 con 0x7f9e4c101000 2024-03-12T03:11:48.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.897+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 >> v1:172.21.15.94:6790/0 conn(0x7f9e4c108c20 legacy=0x7f9e4c1a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:11:48.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.897+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 >> v1:172.21.15.94:6789/0 conn(0x7f9e4c104f50 legacy=0x7f9e4c19d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:11:48.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.897+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4c1a5790 con 0x7f9e4c101000 2024-03-12T03:11:48.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.897+0000 7f9e513bd700 1 -- 172.21.15.151:0/3219219254 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4c1a2380 con 0x7f9e4c101000 2024-03-12T03:11:48.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.897+0000 7f9e513bd700 1 -- 172.21.15.151:0/3219219254 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4c1a28c0 con 0x7f9e4c101000 2024-03-12T03:11:48.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.897+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f9e3c001c40 con 0x7f9e4c101000 2024-03-12T03:11:48.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.897+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f9e3c005cc0 con 0x7f9e4c101000 2024-03-12T03:11:48.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.898+0000 7f9e513bd700 1 -- 172.21.15.151:0/3219219254 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4c0686d0 con 0x7f9e4c101000 2024-03-12T03:11:48.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.898+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 15) v1 ==== 91719+0+0 (unknown 3315827787 0 0) 0x7f9e3c005f90 con 0x7f9e4c101000 2024-03-12T03:11:48.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.899+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 2803+0+0 (unknown 3443377465 0 0) 0x7f9e3c088410 con 0x7f9e4c101000 2024-03-12T03:11:48.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:48.903+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f9e3c005ca0 con 0x7f9e4c101000 2024-03-12T03:11:49.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:11:49.108+0000 7f9e513bd700 1 -- 172.21.15.151:0/3219219254 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e4c1a2bb0 con 0x7f9e3806eb70 2024-03-12T03:11:49.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: Detected new or changed devices on smithi094 2024-03-12T03:11:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: Adjusting osd_memory_target on smithi094 to 4016M 2024-03-12T03:11:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:11:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:11:49.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:49.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:11:49.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:11:49.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:11:49.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:49 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: Detected new or changed devices on smithi094 2024-03-12T03:11:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: Adjusting osd_memory_target on smithi094 to 4016M 2024-03-12T03:11:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:11:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:11:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:49.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:11:49.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:11:49.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:11:49.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:49.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: Detected new or changed devices on smithi094 2024-03-12T03:11:49.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:49.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:49.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:49.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:11:49.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: Adjusting osd_memory_target on smithi094 to 4016M 2024-03-12T03:11:49.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:49.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:11:49.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:49.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:11:49.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:49.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:11:49.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:11:49.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:11:49.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:11:49.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:49 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:50.392 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:50 smithi151 ceph-mon[32923]: from='client.24232 v1:172.21.15.151:0/3219219254' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:11:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:50 smithi094 ceph-mon[29291]: from='client.24232 v1:172.21.15.151:0/3219219254' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:11:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:50 smithi094 ceph-mon[40484]: from='client.24232 v1:172.21.15.151:0/3219219254' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:11:51.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:51 smithi094 ceph-mon[29291]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:51 smithi094 ceph-mon[40484]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:51 smithi151 ceph-mon[32923]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:52.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/2949732951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93461da9-bc47-489d-835e-89fcf88c9287"}]: dispatch 2024-03-12T03:11:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:52 smithi094 ceph-mon[29291]: from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93461da9-bc47-489d-835e-89fcf88c9287"}]: dispatch 2024-03-12T03:11:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:52 smithi094 ceph-mon[29291]: from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93461da9-bc47-489d-835e-89fcf88c9287"}]': finished 2024-03-12T03:11:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:52 smithi094 ceph-mon[29291]: osdmap e27: 5 total, 4 up, 5 in 2024-03-12T03:11:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:52 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:11:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/2949732951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93461da9-bc47-489d-835e-89fcf88c9287"}]: dispatch 2024-03-12T03:11:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:52 smithi094 ceph-mon[40484]: from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93461da9-bc47-489d-835e-89fcf88c9287"}]: dispatch 2024-03-12T03:11:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:52 smithi094 ceph-mon[40484]: from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93461da9-bc47-489d-835e-89fcf88c9287"}]': finished 2024-03-12T03:11:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:52 smithi094 ceph-mon[40484]: osdmap e27: 5 total, 4 up, 5 in 2024-03-12T03:11:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:52 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:11:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/2949732951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93461da9-bc47-489d-835e-89fcf88c9287"}]: dispatch 2024-03-12T03:11:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:52 smithi151 ceph-mon[32923]: from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93461da9-bc47-489d-835e-89fcf88c9287"}]: dispatch 2024-03-12T03:11:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:52 smithi151 ceph-mon[32923]: from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93461da9-bc47-489d-835e-89fcf88c9287"}]': finished 2024-03-12T03:11:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:52 smithi151 ceph-mon[32923]: osdmap e27: 5 total, 4 up, 5 in 2024-03-12T03:11:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:52 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:11:53.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:53 smithi094 ceph-mon[29291]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/1160239746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:11:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:53 smithi094 ceph-mon[40484]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/1160239746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:11:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:53 smithi151 ceph-mon[32923]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/1160239746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:11:55.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:55 smithi151 ceph-mon[32923]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:55.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:55 smithi094 ceph-mon[40484]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:55 smithi094 ceph-mon[29291]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:57.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:57 smithi094 ceph-mon[40484]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:57 smithi094 ceph-mon[29291]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:57 smithi151 ceph-mon[32923]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T03:11:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:58.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T03:11:58.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T03:11:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:11:59.705 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:59 smithi151 ceph-mon[32923]: Deploying daemon osd.4 on smithi151 2024-03-12T03:11:59.705 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:11:59 smithi151 ceph-mon[32923]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:59 smithi094 ceph-mon[29291]: Deploying daemon osd.4 on smithi151 2024-03-12T03:11:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:11:59 smithi094 ceph-mon[29291]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:11:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:59 smithi094 ceph-mon[40484]: Deploying daemon osd.4 on smithi151 2024-03-12T03:11:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:11:59 smithi094 ceph-mon[40484]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:01.630 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:01 smithi151 ceph-mon[32923]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:01.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:01 smithi094 ceph-mon[29291]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:01.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:01 smithi094 ceph-mon[40484]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:03.684 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:03 smithi151 ceph-mon[32923]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:03.685 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:03 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:03.685 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:03 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:03.685 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:03 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:03.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:03 smithi094 ceph-mon[40484]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:03.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:03 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:03.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:03 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:03.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:03 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:03 smithi094 ceph-mon[29291]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:03 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:03 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:03 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:05.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:05 smithi094 ceph-mon[29291]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:05 smithi094 ceph-mon[40484]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:05 smithi151 ceph-mon[32923]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:06.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:06.769+0000 7f9e48ff9700 1 -- 172.21.15.151:0/3219219254 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 1377652169) 0x7f9e4c1a2bb0 con 0x7f9e3806eb70 2024-03-12T03:12:06.773 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 4 on host 'smithi151' 2024-03-12T03:12:06.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:06.771+0000 7f9e513bd700 1 -- 172.21.15.151:0/3219219254 >> v1:172.21.15.94:6800/1299282424 conn(0x7f9e3806eb70 legacy=0x7f9e38071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:12:06.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:06.771+0000 7f9e513bd700 1 -- 172.21.15.151:0/3219219254 >> v1:172.21.15.151:6789/0 conn(0x7f9e4c101000 legacy=0x7f9e4c199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:12:06.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:06.771+0000 7f9e513bd700 1 -- 172.21.15.151:0/3219219254 shutdown_connections 2024-03-12T03:12:06.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:06.771+0000 7f9e513bd700 1 -- 172.21.15.151:0/3219219254 >> 172.21.15.151:0/3219219254 conn(0x7f9e4c0fb800 msgr2=0x7f9e4c10b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:12:06.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:06.772+0000 7f9e513bd700 1 -- 172.21.15.151:0/3219219254 shutdown_connections 2024-03-12T03:12:06.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:06.772+0000 7f9e513bd700 1 -- 172.21.15.151:0/3219219254 wait complete. 2024-03-12T03:12:07.099 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.099 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.100 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:07.100 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:07.100 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.100 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:06 smithi151 ceph-mon[32923]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:07.100 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:06 smithi151 ceph-mon[32923]: from='osd.4 v1:172.21.15.151:6800/2666240674' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:12:07.100 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:06 smithi151 ceph-mon[32923]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:12:07.100 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.100 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.100 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:06 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[29291]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[29291]: from='osd.4 v1:172.21.15.151:6800/2666240674' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:12:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[29291]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:12:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:07.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:07.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:07.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[40484]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:07.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[40484]: from='osd.4 v1:172.21.15.151:6800/2666240674' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:12:07.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[40484]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:12:07.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:07.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:06 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:07.435 DEBUG:teuthology.orchestra.run.smithi151:osd.4> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.4.service 2024-03-12T03:12:07.437 INFO:tasks.cephadm:Deploying osd.5 on smithi151 with /dev/vg_nvme/lv_3... 2024-03-12T03:12:07.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T03:12:08.163 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:07 smithi151 ceph-mon[32923]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T03:12:08.164 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:07 smithi151 ceph-mon[32923]: osdmap e28: 5 total, 4 up, 5 in 2024-03-12T03:12:08.164 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:07 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:12:08.164 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:07 smithi151 ceph-mon[32923]: from='osd.4 v1:172.21.15.151:6800/2666240674' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:08.164 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:07 smithi151 ceph-mon[32923]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:08.164 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:12:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:12:07.950+0000 7f74a7506700 -1 osd.4 0 waiting for initial osdmap 2024-03-12T03:12:08.164 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:12:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:12:07.956+0000 7f74a2af8700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:12:08.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:07 smithi094 ceph-mon[40484]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T03:12:08.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:07 smithi094 ceph-mon[40484]: osdmap e28: 5 total, 4 up, 5 in 2024-03-12T03:12:08.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:07 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:12:08.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:07 smithi094 ceph-mon[40484]: from='osd.4 v1:172.21.15.151:6800/2666240674' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:08.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:07 smithi094 ceph-mon[40484]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:07 smithi094 ceph-mon[29291]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T03:12:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:07 smithi094 ceph-mon[29291]: osdmap e28: 5 total, 4 up, 5 in 2024-03-12T03:12:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:07 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:12:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:07 smithi094 ceph-mon[29291]: from='osd.4 v1:172.21.15.151:6800/2666240674' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:07 smithi094 ceph-mon[29291]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:08.558 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:12:09.136 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:08 smithi151 ceph-mon[32923]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-12T03:12:09.136 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:08 smithi151 ceph-mon[32923]: osdmap e29: 5 total, 4 up, 5 in 2024-03-12T03:12:09.136 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:08 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:12:09.136 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:08 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:12:09.136 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:08 smithi151 ceph-mon[32923]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:09.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:08 smithi094 ceph-mon[29291]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-12T03:12:09.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:08 smithi094 ceph-mon[29291]: osdmap e29: 5 total, 4 up, 5 in 2024-03-12T03:12:09.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:08 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:12:09.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:08 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:12:09.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:08 smithi094 ceph-mon[29291]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:09.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:08 smithi094 ceph-mon[40484]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-12T03:12:09.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:08 smithi094 ceph-mon[40484]: osdmap e29: 5 total, 4 up, 5 in 2024-03-12T03:12:09.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:08 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:12:09.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:08 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:12:09.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:08 smithi094 ceph-mon[40484]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-12T03:12:10.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:09 smithi094 ceph-mon[29291]: purged_snaps scrub starts 2024-03-12T03:12:10.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:09 smithi094 ceph-mon[29291]: purged_snaps scrub ok 2024-03-12T03:12:10.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:09 smithi094 ceph-mon[29291]: osd.4 v1:172.21.15.151:6800/2666240674 boot 2024-03-12T03:12:10.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:09 smithi094 ceph-mon[29291]: osdmap e30: 5 total, 5 up, 5 in 2024-03-12T03:12:10.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:09 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:12:10.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:09 smithi094 ceph-mon[29291]: osdmap e31: 5 total, 5 up, 5 in 2024-03-12T03:12:10.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:09 smithi094 ceph-mon[40484]: purged_snaps scrub starts 2024-03-12T03:12:10.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:09 smithi094 ceph-mon[40484]: purged_snaps scrub ok 2024-03-12T03:12:10.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:09 smithi094 ceph-mon[40484]: osd.4 v1:172.21.15.151:6800/2666240674 boot 2024-03-12T03:12:10.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:09 smithi094 ceph-mon[40484]: osdmap e30: 5 total, 5 up, 5 in 2024-03-12T03:12:10.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:09 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:12:10.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:09 smithi094 ceph-mon[40484]: osdmap e31: 5 total, 5 up, 5 in 2024-03-12T03:12:10.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:09 smithi151 ceph-mon[32923]: purged_snaps scrub starts 2024-03-12T03:12:10.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:09 smithi151 ceph-mon[32923]: purged_snaps scrub ok 2024-03-12T03:12:10.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:09 smithi151 ceph-mon[32923]: osd.4 v1:172.21.15.151:6800/2666240674 boot 2024-03-12T03:12:10.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:09 smithi151 ceph-mon[32923]: osdmap e30: 5 total, 5 up, 5 in 2024-03-12T03:12:10.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:09 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:12:10.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:09 smithi151 ceph-mon[32923]: osdmap e31: 5 total, 5 up, 5 in 2024-03-12T03:12:11.122 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:10 smithi151 ceph-mon[32923]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:11.122 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:10 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:11.123 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:10 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:11.123 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:10 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:12:11.123 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:10 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:11.123 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:10 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:11.123 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:10 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:11.123 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:10 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:11.123 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:10 smithi151 ceph-mon[32923]: osdmap e32: 5 total, 5 up, 5 in 2024-03-12T03:12:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[29291]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:12:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[29291]: osdmap e32: 5 total, 5 up, 5 in 2024-03-12T03:12:11.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[40484]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:11.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:11.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:11.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:12:11.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:11.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:11.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:11.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:11.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:10 smithi094 ceph-mon[40484]: osdmap e32: 5 total, 5 up, 5 in 2024-03-12T03:12:11.473 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T03:12:11.473 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T03:12:11.473 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-03-12T03:12:11.473 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-03-12T03:12:11.474 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114869 s, 913 MB/s 2024-03-12T03:12:11.474 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-03-12T03:12:12.102 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:11 smithi151 ceph-mon[32923]: Detected new or changed devices on smithi151 2024-03-12T03:12:12.102 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:11 smithi151 ceph-mon[32923]: Adjusting osd_memory_target on smithi151 to 17088M 2024-03-12T03:12:12.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi151:vg_nvme/lv_3 2024-03-12T03:12:12.365 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:12:12.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:11 smithi094 ceph-mon[29291]: Detected new or changed devices on smithi151 2024-03-12T03:12:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:11 smithi094 ceph-mon[29291]: Adjusting osd_memory_target on smithi151 to 17088M 2024-03-12T03:12:12.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:11 smithi094 ceph-mon[40484]: Detected new or changed devices on smithi151 2024-03-12T03:12:12.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:11 smithi094 ceph-mon[40484]: Adjusting osd_memory_target on smithi151 to 17088M 2024-03-12T03:12:13.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:12 smithi151 ceph-mon[32923]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T03:12:13.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:12 smithi151 ceph-mon[32923]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:13.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:12 smithi094 ceph-mon[29291]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T03:12:13.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:12 smithi094 ceph-mon[29291]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:12 smithi094 ceph-mon[40484]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T03:12:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:12 smithi094 ceph-mon[40484]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:13.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.988+0000 7f66bedf0700 1 -- 172.21.15.151:0/3891853984 >> v1:172.21.15.151:6789/0 conn(0x7f66b8108c50 legacy=0x7f66b810b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:12:13.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.988+0000 7f66bedf0700 1 -- 172.21.15.151:0/3891853984 shutdown_connections 2024-03-12T03:12:13.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.988+0000 7f66bedf0700 1 -- 172.21.15.151:0/3891853984 >> 172.21.15.151:0/3891853984 conn(0x7f66b80fc0c0 msgr2=0x7f66b80fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:12:13.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.988+0000 7f66bedf0700 1 -- 172.21.15.151:0/3891853984 shutdown_connections 2024-03-12T03:12:13.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.988+0000 7f66bedf0700 1 -- 172.21.15.151:0/3891853984 wait complete. 2024-03-12T03:12:13.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.989+0000 7f66bedf0700 1 Processor -- start 2024-03-12T03:12:13.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.989+0000 7f66bedf0700 1 -- start start 2024-03-12T03:12:13.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.989+0000 7f66bedf0700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66b8197cd0 con 0x7f66b8108c50 2024-03-12T03:12:13.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.989+0000 7f66bedf0700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66b81a34c0 con 0x7f66b8100ae0 2024-03-12T03:12:13.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.989+0000 7f66bedf0700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66b81a46a0 con 0x7f66b8104f10 2024-03-12T03:12:13.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.989+0000 7f66bcb8c700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f66b8100ae0 0x7f66b8197180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:60138/0 (socket says 172.21.15.151:60138) 2024-03-12T03:12:13.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.989+0000 7f66bcb8c700 1 -- 172.21.15.151:0/391408666 learned_addr learned my addr 172.21.15.151:0/391408666 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:12:13.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.989+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685920685 0 0) 0x7f66b81a34c0 con 0x7f66b8100ae0 2024-03-12T03:12:13.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.989+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a80035b0 con 0x7f66b8100ae0 2024-03-12T03:12:13.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18100445 0 0) 0x7f66b81a46a0 con 0x7f66b8104f10 2024-03-12T03:12:13.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b81a34c0 con 0x7f66b8104f10 2024-03-12T03:12:13.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3594833739 0 0) 0x7f66a80035b0 con 0x7f66b8100ae0 2024-03-12T03:12:13.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66b81a46a0 con 0x7f66b8100ae0 2024-03-12T03:12:13.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f66ac001970 con 0x7f66b8100ae0 2024-03-12T03:12:13.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3000440869 0 0) 0x7f66b81a34c0 con 0x7f66b8104f10 2024-03-12T03:12:13.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66a80035b0 con 0x7f66b8104f10 2024-03-12T03:12:13.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f66a4001830 con 0x7f66b8104f10 2024-03-12T03:12:13.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2932252340 0 0) 0x7f66b81a46a0 con 0x7f66b8100ae0 2024-03-12T03:12:13.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 >> v1:172.21.15.94:6790/0 conn(0x7f66b8104f10 legacy=0x7f66b819d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:12:13.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 >> v1:172.21.15.94:6789/0 conn(0x7f66b8108c50 legacy=0x7f66b81a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:12:13.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b81a58c0 con 0x7f66b8100ae0 2024-03-12T03:12:13.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66bedf0700 1 -- 172.21.15.151:0/391408666 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b81a48d0 con 0x7f66b8100ae0 2024-03-12T03:12:13.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f66ac001cc0 con 0x7f66b8100ae0 2024-03-12T03:12:13.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f66ac005d70 con 0x7f66b8100ae0 2024-03-12T03:12:13.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.990+0000 7f66bedf0700 1 -- 172.21.15.151:0/391408666 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b81a4e10 con 0x7f66b8100ae0 2024-03-12T03:12:13.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.991+0000 7f66bedf0700 1 -- 172.21.15.151:0/391408666 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b80686d0 con 0x7f66b8100ae0 2024-03-12T03:12:13.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.992+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 15) v1 ==== 91719+0+0 (unknown 3315827787 0 0) 0x7f66ac003950 con 0x7f66b8100ae0 2024-03-12T03:12:13.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.992+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(34..34 src has 1..34) v4 ==== 3205+0+0 (unknown 1941060179 0 0) 0x7f66ac0886a0 con 0x7f66b8100ae0 2024-03-12T03:12:13.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:13.997+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f66ac056b30 con 0x7f66b8100ae0 2024-03-12T03:12:14.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:14.202+0000 7f66bedf0700 1 -- 172.21.15.151:0/391408666 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f66b81a5100 con 0x7f66a806eb00 2024-03-12T03:12:14.205 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:13 smithi151 ceph-mon[32923]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T03:12:14.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:13 smithi094 ceph-mon[29291]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T03:12:14.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:13 smithi094 ceph-mon[40484]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T03:12:15.139 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:14 smithi151 ceph-mon[32923]: from='client.24259 v1:172.21.15.151:0/391408666' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:12:15.139 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:14 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:12:15.139 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:14 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:12:15.139 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:14 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:15.139 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:14 smithi151 ceph-mon[32923]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:15.361 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:14 smithi094 ceph-mon[29291]: from='client.24259 v1:172.21.15.151:0/391408666' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:12:15.361 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:14 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:12:15.361 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:14 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:12:15.361 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:14 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:15.361 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:14 smithi094 ceph-mon[29291]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:15.362 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:14 smithi094 ceph-mon[40484]: from='client.24259 v1:172.21.15.151:0/391408666' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:12:15.362 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:14 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:12:15.362 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:14 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:12:15.362 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:14 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:15.362 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:14 smithi094 ceph-mon[40484]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[29291]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:12:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/882287119' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c72b1d7-28f6-4ee3-bb80-57ff924abc21"}]: dispatch 2024-03-12T03:12:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[29291]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c72b1d7-28f6-4ee3-bb80-57ff924abc21"}]: dispatch 2024-03-12T03:12:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[29291]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c72b1d7-28f6-4ee3-bb80-57ff924abc21"}]': finished 2024-03-12T03:12:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[29291]: osdmap e35: 6 total, 5 up, 6 in 2024-03-12T03:12:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/1688681228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:12:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[40484]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:12:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/882287119' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c72b1d7-28f6-4ee3-bb80-57ff924abc21"}]: dispatch 2024-03-12T03:12:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[40484]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c72b1d7-28f6-4ee3-bb80-57ff924abc21"}]: dispatch 2024-03-12T03:12:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[40484]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c72b1d7-28f6-4ee3-bb80-57ff924abc21"}]': finished 2024-03-12T03:12:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[40484]: osdmap e35: 6 total, 5 up, 6 in 2024-03-12T03:12:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/1688681228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:12:17.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:17 smithi151 ceph-mon[32923]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:12:17.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/882287119' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c72b1d7-28f6-4ee3-bb80-57ff924abc21"}]: dispatch 2024-03-12T03:12:17.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:17 smithi151 ceph-mon[32923]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c72b1d7-28f6-4ee3-bb80-57ff924abc21"}]: dispatch 2024-03-12T03:12:17.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:17 smithi151 ceph-mon[32923]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c72b1d7-28f6-4ee3-bb80-57ff924abc21"}]': finished 2024-03-12T03:12:17.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:17 smithi151 ceph-mon[32923]: osdmap e35: 6 total, 5 up, 6 in 2024-03-12T03:12:17.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:17 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:17.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/1688681228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:12:19.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:19 smithi094 ceph-mon[29291]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-12T03:12:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:19 smithi094 ceph-mon[40484]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-12T03:12:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:19 smithi151 ceph-mon[32923]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-12T03:12:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:21 smithi094 ceph-mon[29291]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:12:21.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:21 smithi094 ceph-mon[40484]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:12:21.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:21 smithi151 ceph-mon[32923]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:12:23.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:23 smithi094 ceph-mon[29291]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-12T03:12:23.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:23 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T03:12:23.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:23 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:23 smithi094 ceph-mon[40484]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-12T03:12:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:23 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T03:12:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:23 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:23.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:23 smithi151 ceph-mon[32923]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-12T03:12:23.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:23 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T03:12:23.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:23 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:24.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:24 smithi094 ceph-mon[29291]: Deploying daemon osd.5 on smithi151 2024-03-12T03:12:24.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:24 smithi094 ceph-mon[40484]: Deploying daemon osd.5 on smithi151 2024-03-12T03:12:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:24 smithi151 ceph-mon[32923]: Deploying daemon osd.5 on smithi151 2024-03-12T03:12:25.676 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:25 smithi151 ceph-mon[32923]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T03:12:25.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:25 smithi094 ceph-mon[29291]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T03:12:25.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:25 smithi094 ceph-mon[40484]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T03:12:27.701 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:27 smithi151 ceph-mon[32923]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:27.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:27 smithi094 ceph-mon[29291]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:27.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:27 smithi094 ceph-mon[40484]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:29.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:29 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:29.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:29 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:29.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:29 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:29.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:29 smithi151 ceph-mon[32923]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:29.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:29 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:29 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:29 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:29 smithi094 ceph-mon[29291]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:29 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:29 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:29 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:29 smithi094 ceph-mon[40484]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:31.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:31.315+0000 7f66b5ffb700 1 -- 172.21.15.151:0/391408666 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 235447165) 0x7f66b81a5100 con 0x7f66a806eb00 2024-03-12T03:12:31.318 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 5 on host 'smithi151' 2024-03-12T03:12:31.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:31.317+0000 7f66bedf0700 1 -- 172.21.15.151:0/391408666 >> v1:172.21.15.94:6800/1299282424 conn(0x7f66a806eb00 legacy=0x7f66a8070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:12:31.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:31.317+0000 7f66bedf0700 1 -- 172.21.15.151:0/391408666 >> v1:172.21.15.151:6789/0 conn(0x7f66b8100ae0 legacy=0x7f66b8197180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:12:31.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:31.317+0000 7f66bedf0700 1 -- 172.21.15.151:0/391408666 shutdown_connections 2024-03-12T03:12:31.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:31.317+0000 7f66bedf0700 1 -- 172.21.15.151:0/391408666 >> 172.21.15.151:0/391408666 conn(0x7f66b80fc0c0 msgr2=0x7f66b806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:12:31.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:31.317+0000 7f66bedf0700 1 -- 172.21.15.151:0/391408666 shutdown_connections 2024-03-12T03:12:31.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:31.317+0000 7f66bedf0700 1 -- 172.21.15.151:0/391408666 wait complete. 2024-03-12T03:12:31.560 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:31 smithi151 ceph-mon[32923]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:31.560 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:31 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:31.560 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:31 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:31.560 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:31 smithi151 ceph-mon[32923]: from='osd.5 v1:172.21.15.151:6804/2718612978' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:12:31.561 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:31 smithi151 ceph-mon[32923]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:12:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:31 smithi094 ceph-mon[29291]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:31 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:31 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:31 smithi094 ceph-mon[29291]: from='osd.5 v1:172.21.15.151:6804/2718612978' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:12:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:31 smithi094 ceph-mon[29291]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:12:31.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:31 smithi094 ceph-mon[40484]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:31.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:31 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:31.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:31 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:31.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:31 smithi094 ceph-mon[40484]: from='osd.5 v1:172.21.15.151:6804/2718612978' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:12:31.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:31 smithi094 ceph-mon[40484]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:12:32.143 DEBUG:teuthology.orchestra.run.smithi151:osd.5> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.5.service 2024-03-12T03:12:32.145 INFO:tasks.cephadm:Deploying osd.6 on smithi151 with /dev/vg_nvme/lv_2... 2024-03-12T03:12:32.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T03:12:32.428 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:12:33.524 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:33 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:33.524 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:33 smithi151 ceph-mon[32923]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T03:12:33.524 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:33 smithi151 ceph-mon[32923]: osdmap e36: 6 total, 5 up, 6 in 2024-03-12T03:12:33.524 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:33 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:33.524 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:33 smithi151 ceph-mon[32923]: from='osd.5 v1:172.21.15.151:6804/2718612978' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:33.525 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:33 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:33.525 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:33 smithi151 ceph-mon[32923]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:33.525 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:33 smithi151 ceph-mon[32923]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:33.525 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:12:33 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:12:33.296+0000 7fca7faaa700 -1 osd.5 0 waiting for initial osdmap 2024-03-12T03:12:33.525 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:12:33 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:12:33.300+0000 7fca7b89d700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:12:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[29291]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T03:12:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[29291]: osdmap e36: 6 total, 5 up, 6 in 2024-03-12T03:12:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[29291]: from='osd.5 v1:172.21.15.151:6804/2718612978' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[29291]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[29291]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[40484]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T03:12:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[40484]: osdmap e36: 6 total, 5 up, 6 in 2024-03-12T03:12:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[40484]: from='osd.5 v1:172.21.15.151:6804/2718612978' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[40484]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:33 smithi094 ceph-mon[40484]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:12:34.383 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T03:12:34.383 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T03:12:34.383 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-03-12T03:12:34.383 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-03-12T03:12:34.383 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125268 s, 837 MB/s 2024-03-12T03:12:34.383 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-03-12T03:12:34.455 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:34 smithi151 ceph-mon[32923]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-12T03:12:34.455 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:34 smithi151 ceph-mon[32923]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T03:12:34.455 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:34.455 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:34 smithi094 ceph-mon[29291]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-12T03:12:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:34 smithi094 ceph-mon[29291]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T03:12:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:34 smithi094 ceph-mon[40484]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-12T03:12:34.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:34 smithi094 ceph-mon[40484]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T03:12:34.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:34.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:35.556 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:35 smithi151 ceph-mon[32923]: purged_snaps scrub starts 2024-03-12T03:12:35.556 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:35 smithi151 ceph-mon[32923]: purged_snaps scrub ok 2024-03-12T03:12:35.556 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:35 smithi151 ceph-mon[32923]: osd.5 v1:172.21.15.151:6804/2718612978 boot 2024-03-12T03:12:35.556 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:35 smithi151 ceph-mon[32923]: osdmap e38: 6 total, 6 up, 6 in 2024-03-12T03:12:35.556 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:35 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:35.556 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:35 smithi151 ceph-mon[32923]: pgmap v134: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:35 smithi094 ceph-mon[29291]: purged_snaps scrub starts 2024-03-12T03:12:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:35 smithi094 ceph-mon[29291]: purged_snaps scrub ok 2024-03-12T03:12:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:35 smithi094 ceph-mon[29291]: osd.5 v1:172.21.15.151:6804/2718612978 boot 2024-03-12T03:12:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:35 smithi094 ceph-mon[29291]: osdmap e38: 6 total, 6 up, 6 in 2024-03-12T03:12:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:35 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:35 smithi094 ceph-mon[29291]: pgmap v134: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:35 smithi094 ceph-mon[40484]: purged_snaps scrub starts 2024-03-12T03:12:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:35 smithi094 ceph-mon[40484]: purged_snaps scrub ok 2024-03-12T03:12:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:35 smithi094 ceph-mon[40484]: osd.5 v1:172.21.15.151:6804/2718612978 boot 2024-03-12T03:12:35.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:35 smithi094 ceph-mon[40484]: osdmap e38: 6 total, 6 up, 6 in 2024-03-12T03:12:35.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:35 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:12:35.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:35 smithi094 ceph-mon[40484]: pgmap v134: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:36.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi151:vg_nvme/lv_2 2024-03-12T03:12:36.367 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:36 smithi151 ceph-mon[32923]: osdmap e39: 6 total, 6 up, 6 in 2024-03-12T03:12:36.394 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:12:36.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:36 smithi094 ceph-mon[29291]: osdmap e39: 6 total, 6 up, 6 in 2024-03-12T03:12:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:36 smithi094 ceph-mon[40484]: osdmap e39: 6 total, 6 up, 6 in 2024-03-12T03:12:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:37 smithi151 ceph-mon[32923]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T03:12:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:37 smithi151 ceph-mon[32923]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:12:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:12:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[29291]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T03:12:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[29291]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:12:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:12:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[40484]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T03:12:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[40484]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:12:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:12:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:38.395 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:38 smithi151 ceph-mon[32923]: Detected new or changed devices on smithi151 2024-03-12T03:12:38.396 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:38 smithi151 ceph-mon[32923]: Adjusting osd_memory_target on smithi151 to 8544M 2024-03-12T03:12:38.396 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:38 smithi151 ceph-mon[32923]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T03:12:38.396 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:38 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:38.396 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:38 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:38.396 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:38 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:38.396 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:38 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:38.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.404+0000 7f755c782700 1 -- 172.21.15.151:0/4054940602 >> v1:172.21.15.151:6789/0 conn(0x7f7554100ae0 legacy=0x7f7554100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:12:38.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.405+0000 7f755c782700 1 -- 172.21.15.151:0/4054940602 shutdown_connections 2024-03-12T03:12:38.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.405+0000 7f755c782700 1 -- 172.21.15.151:0/4054940602 >> 172.21.15.151:0/4054940602 conn(0x7f75540fc0a0 msgr2=0x7f75540fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:12:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.405+0000 7f755c782700 1 -- 172.21.15.151:0/4054940602 shutdown_connections 2024-03-12T03:12:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.405+0000 7f755c782700 1 -- 172.21.15.151:0/4054940602 wait complete. 2024-03-12T03:12:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.405+0000 7f755c782700 1 Processor -- start 2024-03-12T03:12:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.405+0000 7f755c782700 1 -- start start 2024-03-12T03:12:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.405+0000 7f755c782700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f755406b070 con 0x7f7554108c50 2024-03-12T03:12:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.405+0000 7f755c782700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f75541a3480 con 0x7f7554104f10 2024-03-12T03:12:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.405+0000 7f755c782700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f75541a46c0 con 0x7f7554100ae0 2024-03-12T03:12:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f7559d1d700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f7554104f10 0x7f755419d620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58472/0 (socket says 172.21.15.151:58472) 2024-03-12T03:12:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f755a51e700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f7554100ae0 0x7f7554199ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.151:48660/0 (socket says 172.21.15.151:48660) 2024-03-12T03:12:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f7559d1d700 1 -- 172.21.15.151:0/2219447352 learned_addr learned my addr 172.21.15.151:0/2219447352 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:12:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111862708 0 0) 0x7f75541a3480 con 0x7f7554104f10 2024-03-12T03:12:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75400035b0 con 0x7f7554104f10 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445751854 0 0) 0x7f755406b070 con 0x7f7554108c50 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75541a3480 con 0x7f7554108c50 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366836798 0 0) 0x7f75541a46c0 con 0x7f7554100ae0 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755406b070 con 0x7f7554100ae0 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2251806864 0 0) 0x7f75400035b0 con 0x7f7554104f10 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75541a46c0 con 0x7f7554104f10 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f75440018a0 con 0x7f7554104f10 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1371798476 0 0) 0x7f75541a3480 con 0x7f7554108c50 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75400035b0 con 0x7f7554108c50 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1069940350 0 0) 0x7f755406b070 con 0x7f7554100ae0 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75541a3480 con 0x7f7554100ae0 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f754c001c30 con 0x7f7554108c50 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f7550003860 con 0x7f7554100ae0 2024-03-12T03:12:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3461223926 0 0) 0x7f75541a46c0 con 0x7f7554104f10 2024-03-12T03:12:38.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 >> v1:172.21.15.94:6790/0 conn(0x7f7554100ae0 legacy=0x7f7554199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:12:38.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 >> v1:172.21.15.94:6789/0 conn(0x7f7554108c50 legacy=0x7f75541a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:12:38.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75541a5900 con 0x7f7554104f10 2024-03-12T03:12:38.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f755c782700 1 -- 172.21.15.151:0/2219447352 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75541a24d0 con 0x7f7554104f10 2024-03-12T03:12:38.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f755c782700 1 -- 172.21.15.151:0/2219447352 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75541a2a10 con 0x7f7554104f10 2024-03-12T03:12:38.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f7544003b30 con 0x7f7554104f10 2024-03-12T03:12:38.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.406+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f7544005cc0 con 0x7f7554104f10 2024-03-12T03:12:38.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.407+0000 7f755c782700 1 -- 172.21.15.151:0/2219447352 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7538005150 con 0x7f7554104f10 2024-03-12T03:12:38.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.408+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 15) v1 ==== 91719+0+0 (unknown 3315827787 0 0) 0x7f754401c2f0 con 0x7f7554104f10 2024-03-12T03:12:38.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.408+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(42..42 src has 1..42) v4 ==== 3497+0+0 (unknown 2476507813 0 0) 0x7f7544088620 con 0x7f7554104f10 2024-03-12T03:12:38.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.413+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f7544056970 con 0x7f7554104f10 2024-03-12T03:12:38.617 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:38.614+0000 7f755c782700 1 -- 172.21.15.151:0/2219447352 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7538000bc0 con 0x7f754006ec70 2024-03-12T03:12:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[29291]: Detected new or changed devices on smithi151 2024-03-12T03:12:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[29291]: Adjusting osd_memory_target on smithi151 to 8544M 2024-03-12T03:12:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[29291]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T03:12:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[40484]: Detected new or changed devices on smithi151 2024-03-12T03:12:38.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[40484]: Adjusting osd_memory_target on smithi151 to 8544M 2024-03-12T03:12:38.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[40484]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T03:12:38.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:38.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:38.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:38.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:38 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:39.490 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:39 smithi151 ceph-mon[32923]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T03:12:39.491 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:39 smithi151 ceph-mon[32923]: pgmap v140: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:39.491 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:12:39.491 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:12:39.491 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:39 smithi094 ceph-mon[29291]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T03:12:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:39 smithi094 ceph-mon[29291]: pgmap v140: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:39 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:12:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:39 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:12:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:39 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:39 smithi094 ceph-mon[40484]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T03:12:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:39 smithi094 ceph-mon[40484]: pgmap v140: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:39 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:12:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:39 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:12:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:39 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:40 smithi094 ceph-mon[29291]: from='client.24286 v1:172.21.15.151:0/2219447352' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:12:40.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:40 smithi094 ceph-mon[40484]: from='client.24286 v1:172.21.15.151:0/2219447352' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:12:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:40 smithi151 ceph-mon[32923]: from='client.24286 v1:172.21.15.151:0/2219447352' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:12:41.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:41 smithi094 ceph-mon[29291]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-12T03:12:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/2963606148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d852205d-6b9d-4a4f-83b4-a0e59aa37657"}]: dispatch 2024-03-12T03:12:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:41 smithi094 ceph-mon[29291]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d852205d-6b9d-4a4f-83b4-a0e59aa37657"}]: dispatch 2024-03-12T03:12:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:41 smithi094 ceph-mon[29291]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d852205d-6b9d-4a4f-83b4-a0e59aa37657"}]': finished 2024-03-12T03:12:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:41 smithi094 ceph-mon[29291]: osdmap e43: 7 total, 6 up, 7 in 2024-03-12T03:12:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:41 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:41 smithi094 ceph-mon[40484]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-12T03:12:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/2963606148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d852205d-6b9d-4a4f-83b4-a0e59aa37657"}]: dispatch 2024-03-12T03:12:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:41 smithi094 ceph-mon[40484]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d852205d-6b9d-4a4f-83b4-a0e59aa37657"}]: dispatch 2024-03-12T03:12:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:41 smithi094 ceph-mon[40484]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d852205d-6b9d-4a4f-83b4-a0e59aa37657"}]': finished 2024-03-12T03:12:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:41 smithi094 ceph-mon[40484]: osdmap e43: 7 total, 6 up, 7 in 2024-03-12T03:12:41.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:41 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:41 smithi151 ceph-mon[32923]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-12T03:12:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/2963606148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d852205d-6b9d-4a4f-83b4-a0e59aa37657"}]: dispatch 2024-03-12T03:12:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:41 smithi151 ceph-mon[32923]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d852205d-6b9d-4a4f-83b4-a0e59aa37657"}]: dispatch 2024-03-12T03:12:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:41 smithi151 ceph-mon[32923]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d852205d-6b9d-4a4f-83b4-a0e59aa37657"}]': finished 2024-03-12T03:12:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:41 smithi151 ceph-mon[32923]: osdmap e43: 7 total, 6 up, 7 in 2024-03-12T03:12:41.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:41 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:42.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/3420762370' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:12:42.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/3420762370' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:12:42.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/3420762370' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:12:43.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:43 smithi151 ceph-mon[32923]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:12:43.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:43 smithi094 ceph-mon[29291]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:12:43.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:43 smithi094 ceph-mon[40484]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:12:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:45 smithi151 ceph-mon[32923]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-12T03:12:45.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:45 smithi094 ceph-mon[29291]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-12T03:12:45.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:45 smithi094 ceph-mon[40484]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-12T03:12:47.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:47 smithi094 ceph-mon[29291]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:12:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:47 smithi094 ceph-mon[40484]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:12:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:47 smithi151 ceph-mon[32923]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:12:48.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:48 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T03:12:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:48 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:48.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:48 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T03:12:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:48 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T03:12:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:48 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:49.627 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:49 smithi151 ceph-mon[32923]: Deploying daemon osd.6 on smithi151 2024-03-12T03:12:49.628 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:49 smithi151 ceph-mon[32923]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T03:12:49.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:49 smithi094 ceph-mon[40484]: Deploying daemon osd.6 on smithi151 2024-03-12T03:12:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:49 smithi094 ceph-mon[40484]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T03:12:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:49 smithi094 ceph-mon[29291]: Deploying daemon osd.6 on smithi151 2024-03-12T03:12:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:49 smithi094 ceph-mon[29291]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T03:12:51.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:51 smithi094 ceph-mon[29291]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:51 smithi094 ceph-mon[40484]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:51 smithi151 ceph-mon[32923]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:53.677 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:53 smithi151 ceph-mon[32923]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:53.677 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:53.677 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:53.677 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:53 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:53 smithi094 ceph-mon[29291]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:53 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:53 smithi094 ceph-mon[40484]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:53 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:55.583 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:55 smithi151 ceph-mon[32923]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:55.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:55 smithi094 ceph-mon[29291]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:55 smithi094 ceph-mon[40484]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:56.802 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:56.802 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:56.802 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:56 smithi151 ceph-mon[32923]: from='osd.6 v1:172.21.15.151:6808/1497503104' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:12:56.802 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:56 smithi151 ceph-mon[32923]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:12:56.802 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:56.802 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:56.802 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:56 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:56.802 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:56 smithi151 ceph-mon[32923]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:56.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:56.881+0000 7f754b7fe700 1 -- 172.21.15.151:0/2219447352 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 3929444513) 0x7f7538000bc0 con 0x7f754006ec70 2024-03-12T03:12:56.884 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 6 on host 'smithi151' 2024-03-12T03:12:56.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:56.883+0000 7f755c782700 1 -- 172.21.15.151:0/2219447352 >> v1:172.21.15.94:6800/1299282424 conn(0x7f754006ec70 legacy=0x7f7540071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:12:56.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:56.883+0000 7f755c782700 1 -- 172.21.15.151:0/2219447352 >> v1:172.21.15.151:6789/0 conn(0x7f7554104f10 legacy=0x7f755419d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:12:56.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:56.883+0000 7f755c782700 1 -- 172.21.15.151:0/2219447352 shutdown_connections 2024-03-12T03:12:56.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:56.883+0000 7f755c782700 1 -- 172.21.15.151:0/2219447352 >> 172.21.15.151:0/2219447352 conn(0x7f75540fc0a0 msgr2=0x7f755410b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:12:56.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:56.883+0000 7f755c782700 1 -- 172.21.15.151:0/2219447352 shutdown_connections 2024-03-12T03:12:56.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:12:56.883+0000 7f755c782700 1 -- 172.21.15.151:0/2219447352 wait complete. 2024-03-12T03:12:56.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:56.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:56.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[29291]: from='osd.6 v1:172.21.15.151:6808/1497503104' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:12:56.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[29291]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:12:56.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:56.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:56.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:56.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[29291]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:56.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:56.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:56.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[40484]: from='osd.6 v1:172.21.15.151:6808/1497503104' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:12:56.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[40484]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:12:56.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:12:56.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:12:56.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:56.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:56 smithi094 ceph-mon[40484]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:57.547 DEBUG:teuthology.orchestra.run.smithi151:osd.6> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.6.service 2024-03-12T03:12:57.551 INFO:tasks.cephadm:Deploying osd.7 on smithi151 with /dev/vg_nvme/lv_1... 2024-03-12T03:12:57.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-12T03:12:57.798 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:57 smithi151 ceph-mon[32923]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T03:12:57.798 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:57 smithi151 ceph-mon[32923]: from='osd.6 v1:172.21.15.151:6808/1497503104' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:57.798 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:57 smithi151 ceph-mon[32923]: osdmap e44: 7 total, 6 up, 7 in 2024-03-12T03:12:57.798 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:57 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:57.798 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:57 smithi151 ceph-mon[32923]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:57.798 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:57 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:57.798 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:57 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:57.798 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:57 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[29291]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T03:12:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[29291]: from='osd.6 v1:172.21.15.151:6808/1497503104' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[29291]: osdmap e44: 7 total, 6 up, 7 in 2024-03-12T03:12:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[29291]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:57.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[40484]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T03:12:57.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[40484]: from='osd.6 v1:172.21.15.151:6808/1497503104' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:57.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[40484]: osdmap e44: 7 total, 6 up, 7 in 2024-03-12T03:12:57.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:57.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[40484]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:12:57.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:57.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:12:57.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:57 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:12:58.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:12:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:12:58.098+0000 7f7b958fe700 -1 osd.6 0 waiting for initial osdmap 2024-03-12T03:12:58.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:12:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:12:58.104+0000 7f7b8d6e9700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:12:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:58 smithi094 ceph-mon[40484]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-12T03:12:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:58 smithi094 ceph-mon[40484]: osdmap e45: 7 total, 6 up, 7 in 2024-03-12T03:12:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:58 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:58 smithi094 ceph-mon[40484]: from='osd.6 ' entity='osd.6' 2024-03-12T03:12:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:58 smithi094 ceph-mon[40484]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:58 smithi094 ceph-mon[29291]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-12T03:12:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:58 smithi094 ceph-mon[29291]: osdmap e45: 7 total, 6 up, 7 in 2024-03-12T03:12:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:58 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:58 smithi094 ceph-mon[29291]: from='osd.6 ' entity='osd.6' 2024-03-12T03:12:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:58 smithi094 ceph-mon[29291]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:58 smithi151 ceph-mon[32923]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-12T03:12:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:58 smithi151 ceph-mon[32923]: osdmap e45: 7 total, 6 up, 7 in 2024-03-12T03:12:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:58 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:58 smithi151 ceph-mon[32923]: from='osd.6 ' entity='osd.6' 2024-03-12T03:12:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:58 smithi151 ceph-mon[32923]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:12:59.154 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:12:59.879 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:59 smithi151 ceph-mon[32923]: purged_snaps scrub starts 2024-03-12T03:12:59.879 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:59 smithi151 ceph-mon[32923]: purged_snaps scrub ok 2024-03-12T03:12:59.879 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:59 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:59.880 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:59 smithi151 ceph-mon[32923]: osd.6 v1:172.21.15.151:6808/1497503104 boot 2024-03-12T03:12:59.880 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:59 smithi151 ceph-mon[32923]: osdmap e46: 7 total, 7 up, 7 in 2024-03-12T03:12:59.880 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:12:59 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:59.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:59 smithi094 ceph-mon[40484]: purged_snaps scrub starts 2024-03-12T03:12:59.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:59 smithi094 ceph-mon[40484]: purged_snaps scrub ok 2024-03-12T03:12:59.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:59 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:59.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:59 smithi094 ceph-mon[40484]: osd.6 v1:172.21.15.151:6808/1497503104 boot 2024-03-12T03:12:59.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:59 smithi094 ceph-mon[40484]: osdmap e46: 7 total, 7 up, 7 in 2024-03-12T03:12:59.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:12:59 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:59.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:59 smithi094 ceph-mon[29291]: purged_snaps scrub starts 2024-03-12T03:12:59.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:59 smithi094 ceph-mon[29291]: purged_snaps scrub ok 2024-03-12T03:12:59.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:59 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:12:59.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:59 smithi094 ceph-mon[29291]: osd.6 v1:172.21.15.151:6808/1497503104 boot 2024-03-12T03:12:59.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:59 smithi094 ceph-mon[29291]: osdmap e46: 7 total, 7 up, 7 in 2024-03-12T03:12:59.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:12:59 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:13:01.346 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:01 smithi151 ceph-mon[32923]: osdmap e47: 7 total, 7 up, 7 in 2024-03-12T03:13:01.347 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:01 smithi151 ceph-mon[32923]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:01.347 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:01 smithi151 ceph-mon[32923]: Detected new or changed devices on smithi151 2024-03-12T03:13:01.347 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:01 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:01.347 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:01 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:01.347 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:01 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:01.347 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:01 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:01.347 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:01 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:01.347 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:01 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:01.347 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:01 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:01.347 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:01 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:01.347 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:01 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:01.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[29291]: osdmap e47: 7 total, 7 up, 7 in 2024-03-12T03:13:01.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[29291]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:01.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[29291]: Detected new or changed devices on smithi151 2024-03-12T03:13:01.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:01.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:01.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:01.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:01.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:01.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:01.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:01.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:01.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:01.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[40484]: osdmap e47: 7 total, 7 up, 7 in 2024-03-12T03:13:01.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[40484]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:01.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[40484]: Detected new or changed devices on smithi151 2024-03-12T03:13:01.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:01.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:01.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:01.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:01.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:01.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:01.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:01.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:01.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:01 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-12T03:13:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-12T03:13:01.799 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-03-12T03:13:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-03-12T03:13:01.800 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132498 s, 791 MB/s 2024-03-12T03:13:01.800 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-03-12T03:13:02.409 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:02 smithi151 ceph-mon[32923]: Adjusting osd_memory_target on smithi151 to 5696M 2024-03-12T03:13:02.409 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:02 smithi151 ceph-mon[32923]: osdmap e48: 7 total, 7 up, 7 in 2024-03-12T03:13:02.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi151:vg_nvme/lv_1 2024-03-12T03:13:02.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:02 smithi094 ceph-mon[29291]: Adjusting osd_memory_target on smithi151 to 5696M 2024-03-12T03:13:02.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:02 smithi094 ceph-mon[29291]: osdmap e48: 7 total, 7 up, 7 in 2024-03-12T03:13:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:02 smithi094 ceph-mon[40484]: Adjusting osd_memory_target on smithi151 to 5696M 2024-03-12T03:13:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:02 smithi094 ceph-mon[40484]: osdmap e48: 7 total, 7 up, 7 in 2024-03-12T03:13:02.685 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:13:03.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:03 smithi094 ceph-mon[29291]: osdmap e49: 7 total, 7 up, 7 in 2024-03-12T03:13:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:03 smithi094 ceph-mon[29291]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:03.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:03 smithi094 ceph-mon[40484]: osdmap e49: 7 total, 7 up, 7 in 2024-03-12T03:13:03.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:03 smithi094 ceph-mon[40484]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:03.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:03 smithi151 ceph-mon[32923]: osdmap e49: 7 total, 7 up, 7 in 2024-03-12T03:13:03.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:03 smithi151 ceph-mon[32923]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:04.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.303+0000 7fd6d1635700 1 -- 172.21.15.151:0/826071168 >> v1:172.21.15.151:6789/0 conn(0x7fd6cc104f30 legacy=0x7fd6cc107380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:04.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.303+0000 7fd6d1635700 1 -- 172.21.15.151:0/826071168 shutdown_connections 2024-03-12T03:13:04.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.303+0000 7fd6d1635700 1 -- 172.21.15.151:0/826071168 >> 172.21.15.151:0/826071168 conn(0x7fd6cc0fc0a0 msgr2=0x7fd6cc0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:04.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.303+0000 7fd6d1635700 1 -- 172.21.15.151:0/826071168 shutdown_connections 2024-03-12T03:13:04.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.303+0000 7fd6d1635700 1 -- 172.21.15.151:0/826071168 wait complete. 2024-03-12T03:13:04.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.304+0000 7fd6d1635700 1 Processor -- start 2024-03-12T03:13:04.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.304+0000 7fd6d1635700 1 -- start start 2024-03-12T03:13:04.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.304+0000 7fd6d1635700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd6cc19a270 con 0x7fd6cc104f30 2024-03-12T03:13:04.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.304+0000 7fd6d1635700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd6cc1a5770 con 0x7fd6cc108c70 2024-03-12T03:13:04.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.304+0000 7fd6d1635700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd6cc1a6950 con 0x7fd6cc100b00 2024-03-12T03:13:04.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.304+0000 7fd6cb7fe700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd6cc108c70 0x7fd6cc1a3040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:52728/0 (socket says 172.21.15.151:52728) 2024-03-12T03:13:04.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.304+0000 7fd6cb7fe700 1 -- 172.21.15.151:0/3262103877 learned_addr learned my addr 172.21.15.151:0/3262103877 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:13:04.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156104345 0 0) 0x7fd6cc1a5770 con 0x7fd6cc108c70 2024-03-12T03:13:04.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b80035b0 con 0x7fd6cc108c70 2024-03-12T03:13:04.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495086242 0 0) 0x7fd6cc19a270 con 0x7fd6cc104f30 2024-03-12T03:13:04.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6cc1a5770 con 0x7fd6cc104f30 2024-03-12T03:13:04.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509731522 0 0) 0x7fd6cc1a6950 con 0x7fd6cc100b00 2024-03-12T03:13:04.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6cc19a270 con 0x7fd6cc100b00 2024-03-12T03:13:04.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1567827696 0 0) 0x7fd6b80035b0 con 0x7fd6cc108c70 2024-03-12T03:13:04.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6cc1a6950 con 0x7fd6cc108c70 2024-03-12T03:13:04.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fd6c4001c70 con 0x7fd6cc108c70 2024-03-12T03:13:04.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 501851321 0 0) 0x7fd6cc1a5770 con 0x7fd6cc104f30 2024-03-12T03:13:04.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6b80035b0 con 0x7fd6cc104f30 2024-03-12T03:13:04.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fd6b4001430 con 0x7fd6cc104f30 2024-03-12T03:13:04.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2163190247 0 0) 0x7fd6cc19a270 con 0x7fd6cc100b00 2024-03-12T03:13:04.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6cc1a5770 con 0x7fd6cc100b00 2024-03-12T03:13:04.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fd6bc001720 con 0x7fd6cc100b00 2024-03-12T03:13:04.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 114724284 0 0) 0x7fd6cc1a6950 con 0x7fd6cc108c70 2024-03-12T03:13:04.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 >> v1:172.21.15.94:6790/0 conn(0x7fd6cc100b00 legacy=0x7fd6cc199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:04.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 >> v1:172.21.15.94:6789/0 conn(0x7fd6cc104f30 legacy=0x7fd6cc19f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:04.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc1a7b30 con 0x7fd6cc108c70 2024-03-12T03:13:04.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6d1635700 1 -- 172.21.15.151:0/3262103877 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6cc1a47c0 con 0x7fd6cc108c70 2024-03-12T03:13:04.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6d1635700 1 -- 172.21.15.151:0/3262103877 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6cc1a4d00 con 0x7fd6cc108c70 2024-03-12T03:13:04.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fd6c4003eb0 con 0x7fd6cc108c70 2024-03-12T03:13:04.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.305+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fd6c4006040 con 0x7fd6cc108c70 2024-03-12T03:13:04.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.306+0000 7fd6d1635700 1 -- 172.21.15.151:0/3262103877 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ac005120 con 0x7fd6cc108c70 2024-03-12T03:13:04.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.307+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 15) v1 ==== 91719+0+0 (unknown 3315827787 0 0) 0x7fd6c4003a20 con 0x7fd6cc108c70 2024-03-12T03:13:04.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.307+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(50..50 src has 1..50) v4 ==== 3789+0+0 (unknown 3100611363 0 0) 0x7fd6c4088ae0 con 0x7fd6cc108c70 2024-03-12T03:13:04.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.312+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fd6c4056cd0 con 0x7fd6cc108c70 2024-03-12T03:13:04.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:04 smithi094 ceph-mon[29291]: osdmap e50: 7 total, 7 up, 7 in 2024-03-12T03:13:04.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:04 smithi094 ceph-mon[40484]: osdmap e50: 7 total, 7 up, 7 in 2024-03-12T03:13:04.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:04 smithi151 ceph-mon[32923]: osdmap e50: 7 total, 7 up, 7 in 2024-03-12T03:13:04.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:04.517+0000 7fd6d1635700 1 -- 172.21.15.151:0/3262103877 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6ac000b90 con 0x7fd6b806ec40 2024-03-12T03:13:05.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:05 smithi094 ceph-mon[29291]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:05.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:05 smithi094 ceph-mon[29291]: from='client.24313 v1:172.21.15.151:0/3262103877' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:05.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:05 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:13:05.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:05 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:13:05.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:05 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:05.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:05 smithi094 ceph-mon[40484]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:05.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:05 smithi094 ceph-mon[40484]: from='client.24313 v1:172.21.15.151:0/3262103877' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:05.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:05 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:13:05.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:05 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:13:05.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:05 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:05.463 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:05 smithi151 ceph-mon[32923]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:05.463 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:05 smithi151 ceph-mon[32923]: from='client.24313 v1:172.21.15.151:0/3262103877' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:05.463 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:05 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:13:05.463 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:05 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:13:05.463 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:05 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:07 smithi094 ceph-mon[29291]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:13:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/2656007210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd9fca37-a6f2-46fc-90de-cd3790ad3331"}]: dispatch 2024-03-12T03:13:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:07 smithi094 ceph-mon[29291]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd9fca37-a6f2-46fc-90de-cd3790ad3331"}]: dispatch 2024-03-12T03:13:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:07 smithi094 ceph-mon[29291]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd9fca37-a6f2-46fc-90de-cd3790ad3331"}]': finished 2024-03-12T03:13:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:07 smithi094 ceph-mon[29291]: osdmap e51: 8 total, 7 up, 8 in 2024-03-12T03:13:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:07 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:07.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:07 smithi094 ceph-mon[40484]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:13:07.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/2656007210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd9fca37-a6f2-46fc-90de-cd3790ad3331"}]: dispatch 2024-03-12T03:13:07.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:07 smithi094 ceph-mon[40484]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd9fca37-a6f2-46fc-90de-cd3790ad3331"}]: dispatch 2024-03-12T03:13:07.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:07 smithi094 ceph-mon[40484]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd9fca37-a6f2-46fc-90de-cd3790ad3331"}]': finished 2024-03-12T03:13:07.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:07 smithi094 ceph-mon[40484]: osdmap e51: 8 total, 7 up, 8 in 2024-03-12T03:13:07.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:07 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:07.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:07 smithi151 ceph-mon[32923]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:13:07.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/2656007210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd9fca37-a6f2-46fc-90de-cd3790ad3331"}]: dispatch 2024-03-12T03:13:07.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:07 smithi151 ceph-mon[32923]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd9fca37-a6f2-46fc-90de-cd3790ad3331"}]: dispatch 2024-03-12T03:13:07.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:07 smithi151 ceph-mon[32923]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd9fca37-a6f2-46fc-90de-cd3790ad3331"}]': finished 2024-03-12T03:13:07.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:07 smithi151 ceph-mon[32923]: osdmap e51: 8 total, 7 up, 8 in 2024-03-12T03:13:07.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:07 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:08.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/4081039785' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:13:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/4081039785' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:13:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/4081039785' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:13:09.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:09 smithi094 ceph-mon[29291]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-12T03:13:09.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:09 smithi094 ceph-mon[40484]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-12T03:13:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:09 smithi151 ceph-mon[32923]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-12T03:13:11.669 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:11 smithi151 ceph-mon[32923]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:13:11.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:11 smithi094 ceph-mon[29291]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:13:11.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:11 smithi094 ceph-mon[40484]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:13:13.680 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:13 smithi151 ceph-mon[32923]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-12T03:13:13.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:13 smithi094 ceph-mon[29291]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-12T03:13:13.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:13 smithi094 ceph-mon[40484]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-12T03:13:14.617 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:14 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T03:13:14.617 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:14 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:14.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:14 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T03:13:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:14 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:14 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T03:13:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:14 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:15.630 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:15 smithi151 ceph-mon[32923]: Deploying daemon osd.7 on smithi151 2024-03-12T03:13:15.630 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:15 smithi151 ceph-mon[32923]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T03:13:15.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:15 smithi094 ceph-mon[29291]: Deploying daemon osd.7 on smithi151 2024-03-12T03:13:15.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:15 smithi094 ceph-mon[29291]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T03:13:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:15 smithi094 ceph-mon[40484]: Deploying daemon osd.7 on smithi151 2024-03-12T03:13:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:15 smithi094 ceph-mon[40484]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T03:13:17.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:17 smithi094 ceph-mon[40484]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:17 smithi094 ceph-mon[29291]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:17 smithi151 ceph-mon[32923]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:19 smithi094 ceph-mon[29291]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:19 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:19 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:19 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:19 smithi094 ceph-mon[40484]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:19 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:19 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:19 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:19.712 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:19 smithi151 ceph-mon[32923]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:19.712 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:19 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:19.712 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:19 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:19.713 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:19 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:21.643 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:21 smithi151 ceph-mon[32923]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:21.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:21 smithi094 ceph-mon[40484]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:21 smithi094 ceph-mon[29291]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:22.532 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:22.529+0000 7fd6c8ff9700 1 -- 172.21.15.151:0/3262103877 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 3055779349) 0x7fd6ac000b90 con 0x7fd6b806ec40 2024-03-12T03:13:22.532 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 7 on host 'smithi151' 2024-03-12T03:13:22.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:22.531+0000 7fd6d1635700 1 -- 172.21.15.151:0/3262103877 >> v1:172.21.15.94:6800/1299282424 conn(0x7fd6b806ec40 legacy=0x7fd6b8071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:22.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:22.531+0000 7fd6d1635700 1 -- 172.21.15.151:0/3262103877 >> v1:172.21.15.151:6789/0 conn(0x7fd6cc108c70 legacy=0x7fd6cc1a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:22.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:22.532+0000 7fd6d1635700 1 -- 172.21.15.151:0/3262103877 shutdown_connections 2024-03-12T03:13:22.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:22.532+0000 7fd6d1635700 1 -- 172.21.15.151:0/3262103877 >> 172.21.15.151:0/3262103877 conn(0x7fd6cc0fc0a0 msgr2=0x7fd6cc06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:22.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:22.532+0000 7fd6d1635700 1 -- 172.21.15.151:0/3262103877 shutdown_connections 2024-03-12T03:13:22.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:22.532+0000 7fd6d1635700 1 -- 172.21.15.151:0/3262103877 wait complete. 2024-03-12T03:13:22.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:22 smithi094 ceph-mon[29291]: from='osd.7 v1:172.21.15.151:6812/683127336' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:13:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:22 smithi094 ceph-mon[29291]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:13:22.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:22 smithi094 ceph-mon[40484]: from='osd.7 v1:172.21.15.151:6812/683127336' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:13:22.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:22 smithi094 ceph-mon[40484]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:13:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:22 smithi151 ceph-mon[32923]: from='osd.7 v1:172.21.15.151:6812/683127336' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:13:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:22 smithi151 ceph-mon[32923]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:13:23.144 DEBUG:teuthology.orchestra.run.smithi151:osd.7> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.7.service 2024-03-12T03:13:23.146 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-12T03:13:23.147 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-12T03:13:23.412 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:13:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[40484]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[40484]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T03:13:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[40484]: osdmap e52: 8 total, 7 up, 8 in 2024-03-12T03:13:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[40484]: from='osd.7 v1:172.21.15.151:6812/683127336' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:13:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[40484]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:13:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:23.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:23.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:23.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[29291]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:23.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[29291]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T03:13:23.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[29291]: osdmap e52: 8 total, 7 up, 8 in 2024-03-12T03:13:23.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:23.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[29291]: from='osd.7 v1:172.21.15.151:6812/683127336' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:13:23.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[29291]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:13:23.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:23.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:23.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:23.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:23 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:23.714 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:13:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:13:23.460+0000 7f271714f700 -1 osd.7 0 waiting for initial osdmap 2024-03-12T03:13:23.714 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:13:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:13:23.465+0000 7f2712f42700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:13:23.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:23 smithi151 ceph-mon[32923]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:23.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:23 smithi151 ceph-mon[32923]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T03:13:23.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:23 smithi151 ceph-mon[32923]: osdmap e52: 8 total, 7 up, 8 in 2024-03-12T03:13:23.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:23 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:23.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:23 smithi151 ceph-mon[32923]: from='osd.7 v1:172.21.15.151:6812/683127336' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:13:23.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:23 smithi151 ceph-mon[32923]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-12T03:13:23.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:23 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:23.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:23 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:23.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:23 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:23.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:23 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:24.671 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:24 smithi094 ceph-mon[29291]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-12T03:13:24.671 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:24 smithi094 ceph-mon[29291]: osdmap e53: 8 total, 7 up, 8 in 2024-03-12T03:13:24.671 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:24 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:24.671 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:24 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:24.671 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:24 smithi094 ceph-mon[40484]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-12T03:13:24.671 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:24 smithi094 ceph-mon[40484]: osdmap e53: 8 total, 7 up, 8 in 2024-03-12T03:13:24.671 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:24 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:24.671 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:24 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:24 smithi151 ceph-mon[32923]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-12T03:13:24.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:24 smithi151 ceph-mon[32923]: osdmap e53: 8 total, 7 up, 8 in 2024-03-12T03:13:24.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:24 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:24.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:24 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:25.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.144+0000 7f6ef369e700 1 -- 172.21.15.94:0/1087902388 >> v1:172.21.15.94:6789/0 conn(0x7f6eec106470 legacy=0x7f6eec1129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:25.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.145+0000 7f6ef369e700 1 -- 172.21.15.94:0/1087902388 shutdown_connections 2024-03-12T03:13:25.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.145+0000 7f6ef369e700 1 -- 172.21.15.94:0/1087902388 >> 172.21.15.94:0/1087902388 conn(0x7f6eec0fc080 msgr2=0x7f6eec0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:25.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.145+0000 7f6ef369e700 1 -- 172.21.15.94:0/1087902388 shutdown_connections 2024-03-12T03:13:25.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.145+0000 7f6ef369e700 1 -- 172.21.15.94:0/1087902388 wait complete. 2024-03-12T03:13:25.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.145+0000 7f6ef369e700 1 Processor -- start 2024-03-12T03:13:25.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.145+0000 7f6ef369e700 1 -- start start 2024-03-12T03:13:25.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.145+0000 7f6ef369e700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6eec19e150 con 0x7f6eec103fb0 2024-03-12T03:13:25.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.145+0000 7f6ef369e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6eec1a9940 con 0x7f6eec106470 2024-03-12T03:13:25.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.145+0000 7f6ef369e700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6eec1aab20 con 0x7f6eec100460 2024-03-12T03:13:25.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.145+0000 7f6ef0c39700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6eec103fb0 0x7f6eec1a3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:40944/0 (socket says 172.21.15.94:40944) 2024-03-12T03:13:25.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.145+0000 7f6ef0c39700 1 -- 172.21.15.94:0/3486352611 learned_addr learned my addr 172.21.15.94:0/3486352611 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:13:25.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.145+0000 7f6ef143a700 1 --1- 172.21.15.94:0/3486352611 >> v1:172.21.15.94:6790/0 conn(0x7f6eec100460 0x7f6eec19d600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:36104/0 (socket says 172.21.15.94:36104) 2024-03-12T03:13:25.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822877862 0 0) 0x7f6eec19e150 con 0x7f6eec103fb0 2024-03-12T03:13:25.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed80035b0 con 0x7f6eec103fb0 2024-03-12T03:13:25.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526797219 0 0) 0x7f6eec1aab20 con 0x7f6eec100460 2024-03-12T03:13:25.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eec19e150 con 0x7f6eec100460 2024-03-12T03:13:25.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241446852 0 0) 0x7f6eec1a9940 con 0x7f6eec106470 2024-03-12T03:13:25.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eec1aab20 con 0x7f6eec106470 2024-03-12T03:13:25.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1721196438 0 0) 0x7f6eec19e150 con 0x7f6eec100460 2024-03-12T03:13:25.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6eec1a9940 con 0x7f6eec100460 2024-03-12T03:13:25.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1345056884 0 0) 0x7f6ed80035b0 con 0x7f6eec103fb0 2024-03-12T03:13:25.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6eec19e150 con 0x7f6eec103fb0 2024-03-12T03:13:25.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f6ee8001820 con 0x7f6eec100460 2024-03-12T03:13:25.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f6edc0013d0 con 0x7f6eec103fb0 2024-03-12T03:13:25.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3717644933 0 0) 0x7f6eec1aab20 con 0x7f6eec106470 2024-03-12T03:13:25.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ed80035b0 con 0x7f6eec106470 2024-03-12T03:13:25.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f6ee4004640 con 0x7f6eec106470 2024-03-12T03:13:25.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 418292900 0 0) 0x7f6eec1a9940 con 0x7f6eec100460 2024-03-12T03:13:25.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 >> v1:172.21.15.151:6789/0 conn(0x7f6eec106470 legacy=0x7f6eec1a7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:25.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 >> v1:172.21.15.94:6789/0 conn(0x7f6eec103fb0 legacy=0x7f6eec1a3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:25.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec1abd40 con 0x7f6eec100460 2024-03-12T03:13:25.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ef369e700 1 -- 172.21.15.94:0/3486352611 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eec1aad50 con 0x7f6eec100460 2024-03-12T03:13:25.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ef369e700 1 -- 172.21.15.94:0/3486352611 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eec1ab290 con 0x7f6eec100460 2024-03-12T03:13:25.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f6ee8001c50 con 0x7f6eec100460 2024-03-12T03:13:25.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.146+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f6ee8005c00 con 0x7f6eec100460 2024-03-12T03:13:25.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.147+0000 7f6ef369e700 1 -- 172.21.15.94:0/3486352611 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed0005150 con 0x7f6eec100460 2024-03-12T03:13:25.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.148+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 15) v1 ==== 91719+0+0 (unknown 3315827787 0 0) 0x7f6ee8003950 con 0x7f6eec100460 2024-03-12T03:13:25.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.148+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(54..54 src has 1..54) v4 ==== 4114+0+0 (unknown 693005573 0 0) 0x7f6ee80885b0 con 0x7f6eec100460 2024-03-12T03:13:25.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.153+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f6ee80567b0 con 0x7f6eec100460 2024-03-12T03:13:25.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.355+0000 7f6ef369e700 1 -- 172.21.15.94:0/3486352611 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f6ed0005440 con 0x7f6eec100460 2024-03-12T03:13:25.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.355+0000 7f6ee27fc700 1 -- 172.21.15.94:0/3486352611 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v54) v1 ==== 74+0+130 (unknown 4107983022 0 2967937705) 0x7f6ee80560b0 con 0x7f6eec100460 2024-03-12T03:13:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:13:25.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.357+0000 7f6ef369e700 1 -- 172.21.15.94:0/3486352611 >> v1:172.21.15.94:6800/1299282424 conn(0x7f6ed806eb40 legacy=0x7f6ed8071000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:25.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.357+0000 7f6ef369e700 1 -- 172.21.15.94:0/3486352611 >> v1:172.21.15.94:6790/0 conn(0x7f6eec100460 legacy=0x7f6eec19d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:25.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.357+0000 7f6ef369e700 1 -- 172.21.15.94:0/3486352611 shutdown_connections 2024-03-12T03:13:25.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.357+0000 7f6ef369e700 1 -- 172.21.15.94:0/3486352611 >> 172.21.15.94:0/3486352611 conn(0x7f6eec0fc080 msgr2=0x7f6eec102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:25.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.357+0000 7f6ef369e700 1 -- 172.21.15.94:0/3486352611 shutdown_connections 2024-03-12T03:13:25.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:25.357+0000 7f6ef369e700 1 -- 172.21.15.94:0/3486352611 wait complete. 2024-03-12T03:13:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:25 smithi151 ceph-mon[32923]: purged_snaps scrub starts 2024-03-12T03:13:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:25 smithi151 ceph-mon[32923]: purged_snaps scrub ok 2024-03-12T03:13:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:25 smithi151 ceph-mon[32923]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:25 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:25 smithi151 ceph-mon[32923]: osd.7 v1:172.21.15.151:6812/683127336 boot 2024-03-12T03:13:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:25 smithi151 ceph-mon[32923]: osdmap e54: 8 total, 8 up, 8 in 2024-03-12T03:13:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:25 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3486352611' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:13:25.738 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[40484]: purged_snaps scrub starts 2024-03-12T03:13:25.738 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[40484]: purged_snaps scrub ok 2024-03-12T03:13:25.738 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[40484]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:25.738 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:25.738 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[40484]: osd.7 v1:172.21.15.151:6812/683127336 boot 2024-03-12T03:13:25.738 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[40484]: osdmap e54: 8 total, 8 up, 8 in 2024-03-12T03:13:25.738 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:25.738 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3486352611' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:13:25.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[29291]: purged_snaps scrub starts 2024-03-12T03:13:25.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[29291]: purged_snaps scrub ok 2024-03-12T03:13:25.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[29291]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:13:25.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:25.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[29291]: osd.7 v1:172.21.15.151:6812/683127336 boot 2024-03-12T03:13:25.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[29291]: osdmap e54: 8 total, 8 up, 8 in 2024-03-12T03:13:25.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:13:25.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3486352611' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:13:26.139 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1710213204,"num_in_osds":8,"osd_in_since":1710213187,"num_remapped_pgs":1} 2024-03-12T03:13:26.139 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:13:26.385 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:13:26.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:26 smithi094 ceph-mon[29291]: osdmap e55: 8 total, 8 up, 8 in 2024-03-12T03:13:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:26 smithi094 ceph-mon[40484]: osdmap e55: 8 total, 8 up, 8 in 2024-03-12T03:13:26.798 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:26 smithi151 ceph-mon[32923]: osdmap e55: 8 total, 8 up, 8 in 2024-03-12T03:13:27.846 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:13:27.846 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: osdmap e56: 8 total, 8 up, 8 in 2024-03-12T03:13:27.846 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.846 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.846 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:27.846 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:27.846 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:27.846 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:27.846 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.846 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:27.847 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:27.847 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.847 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:27.847 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:27.847 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:27.847 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:13:27.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: osdmap e56: 8 total, 8 up, 8 in 2024-03-12T03:13:27.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:27.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:27.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:27.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:27.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:27.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:27.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:27.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:27.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:27.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:27 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:13:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: osdmap e56: 8 total, 8 up, 8 in 2024-03-12T03:13:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:13:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:27 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:28.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.256+0000 7f3fbb03c700 1 -- 172.21.15.94:0/459029483 >> v1:172.21.15.94:6789/0 conn(0x7f3fb410cc80 legacy=0x7f3fb410f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:28.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.256+0000 7f3fbb03c700 1 -- 172.21.15.94:0/459029483 shutdown_connections 2024-03-12T03:13:28.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.256+0000 7f3fbb03c700 1 -- 172.21.15.94:0/459029483 >> 172.21.15.94:0/459029483 conn(0x7f3fb4076ec0 msgr2=0x7f3fb40fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:28.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.256+0000 7f3fbb03c700 1 -- 172.21.15.94:0/459029483 shutdown_connections 2024-03-12T03:13:28.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.256+0000 7f3fbb03c700 1 -- 172.21.15.94:0/459029483 wait complete. 2024-03-12T03:13:28.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.257+0000 7f3fbb03c700 1 Processor -- start 2024-03-12T03:13:28.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.257+0000 7f3fbb03c700 1 -- start start 2024-03-12T03:13:28.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.257+0000 7f3fbb03c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3fb41a8e80 con 0x7f3fb410cc80 2024-03-12T03:13:28.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.257+0000 7f3fbb03c700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3fb41aa0a0 con 0x7f3fb41090e0 2024-03-12T03:13:28.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.257+0000 7f3fbb03c700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3fb41ab2e0 con 0x7f3fb4100450 2024-03-12T03:13:28.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.257+0000 7f3fb95d9700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3fb410cc80 0x7f3fb41a7580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:40960/0 (socket says 172.21.15.94:40960) 2024-03-12T03:13:28.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.257+0000 7f3fb95d9700 1 -- 172.21.15.94:0/2287402302 learned_addr learned my addr 172.21.15.94:0/2287402302 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:13:28.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.257+0000 7f3fb8dd8700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f3fb4100450 0x7f3fb41041a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:36110/0 (socket says 172.21.15.94:36110) 2024-03-12T03:13:28.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.257+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967302157 0 0) 0x7f3fb41a8e80 con 0x7f3fb410cc80 2024-03-12T03:13:28.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.257+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa40046c0 con 0x7f3fb410cc80 2024-03-12T03:13:28.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.257+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948692576 0 0) 0x7f3fb41ab2e0 con 0x7f3fb4100450 2024-03-12T03:13:28.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.257+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fb41a8e80 con 0x7f3fb4100450 2024-03-12T03:13:28.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783297909 0 0) 0x7f3fb41aa0a0 con 0x7f3fb41090e0 2024-03-12T03:13:28.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fb41ab2e0 con 0x7f3fb41090e0 2024-03-12T03:13:28.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 540515603 0 0) 0x7f3fa40046c0 con 0x7f3fb410cc80 2024-03-12T03:13:28.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fb41aa0a0 con 0x7f3fb410cc80 2024-03-12T03:13:28.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f3fac0047c0 con 0x7f3fb410cc80 2024-03-12T03:13:28.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1882377065 0 0) 0x7f3fb41a8e80 con 0x7f3fb4100450 2024-03-12T03:13:28.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fa40046c0 con 0x7f3fb4100450 2024-03-12T03:13:28.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f3fa80017b0 con 0x7f3fb4100450 2024-03-12T03:13:28.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2483759339 0 0) 0x7f3fb41aa0a0 con 0x7f3fb410cc80 2024-03-12T03:13:28.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 >> v1:172.21.15.94:6790/0 conn(0x7f3fb4100450 legacy=0x7f3fb41041a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:28.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 >> v1:172.21.15.151:6789/0 conn(0x7f3fb41090e0 legacy=0x7f3fb41048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:28.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb41ac520 con 0x7f3fb410cc80 2024-03-12T03:13:28.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fbb03c700 1 -- 172.21.15.94:0/2287402302 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb41ab510 con 0x7f3fb410cc80 2024-03-12T03:13:28.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f3fac001e30 con 0x7f3fb410cc80 2024-03-12T03:13:28.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f3fac005330 con 0x7f3fb410cc80 2024-03-12T03:13:28.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.258+0000 7f3fbb03c700 1 -- 172.21.15.94:0/2287402302 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb41ab9d0 con 0x7f3fb410cc80 2024-03-12T03:13:28.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.259+0000 7f3fbb03c700 1 -- 172.21.15.94:0/2287402302 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb40686d0 con 0x7f3fb410cc80 2024-03-12T03:13:28.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.260+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 15) v1 ==== 91719+0+0 (unknown 3315827787 0 0) 0x7f3fac005310 con 0x7f3fb410cc80 2024-03-12T03:13:28.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.260+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(57..57 src has 1..57) v4 ==== 4081+0+0 (unknown 2745960438 0 0) 0x7f3fac088c40 con 0x7f3fb410cc80 2024-03-12T03:13:28.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.265+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f3fac056d00 con 0x7f3fb410cc80 2024-03-12T03:13:28.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.462+0000 7f3fbb03c700 1 -- 172.21.15.94:0/2287402302 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fb41121e0 con 0x7f3fb410cc80 2024-03-12T03:13:28.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.462+0000 7f3fb1ffb700 1 -- 172.21.15.94:0/2287402302 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+11501 (unknown 2588859899 0 1777349585) 0x7f3fac001fe0 con 0x7f3fb410cc80 2024-03-12T03:13:28.465 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:13:28.466 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":57,"fsid":"964142e8-e01d-11ee-95c8-87774f69a715","created":"2024-03-12T03:08:07.892137+0000","modified":"2024-03-12T03:13:27.471267+0000","last_up_change":"2024-03-12T03:13:24.457659+0000","last_in_change":"2024-03-12T03:13:07.180424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:11:20.645672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"003c53a3-baa3-4abc-bbed-e8a0c0db7ab5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":2935792501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":2935792501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":2935792501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":2935792501}]},"public_addr":"172.21.15.94:6801/2935792501","cluster_addr":"172.21.15.94:6802/2935792501","heartbeat_back_addr":"172.21.15.94:6804/2935792501","heartbeat_front_addr":"172.21.15.94:6803/2935792501","state":["exists","up"]},{"osd":1,"uuid":"91c8e6b7-d57f-492b-bc40-9f46769a1c0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":1056655004}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":1056655004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":1056655004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":1056655004}]},"public_addr":"172.21.15.94:6805/1056655004","cluster_addr":"172.21.15.94:6806/1056655004","heartbeat_back_addr":"172.21.15.94:6808/1056655004","heartbeat_front_addr":"172.21.15.94:6807/1056655004","state":["exists","up"]},{"osd":2,"uuid":"4b0ab939-b3de-4821-9c65-348f5172981d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":2929979688}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":2929979688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":2929979688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":2929979688}]},"public_addr":"172.21.15.94:6809/2929979688","cluster_addr":"172.21.15.94:6810/2929979688","heartbeat_back_addr":"172.21.15.94:6812/2929979688","heartbeat_front_addr":"172.21.15.94:6811/2929979688","state":["exists","up"]},{"osd":3,"uuid":"3fd5c9c8-6304-4233-a4df-8c4c4bda5f79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":4235334318}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":4235334318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6816","nonce":4235334318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":4235334318}]},"public_addr":"172.21.15.94:6813/4235334318","cluster_addr":"172.21.15.94:6814/4235334318","heartbeat_back_addr":"172.21.15.94:6816/4235334318","heartbeat_front_addr":"172.21.15.94:6815/4235334318","state":["exists","up"]},{"osd":4,"uuid":"93461da9-bc47-489d-835e-89fcf88c9287","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6800","nonce":2666240674}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6801","nonce":2666240674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6803","nonce":2666240674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6802","nonce":2666240674}]},"public_addr":"172.21.15.151:6800/2666240674","cluster_addr":"172.21.15.151:6801/2666240674","heartbeat_back_addr":"172.21.15.151:6803/2666240674","heartbeat_front_addr":"172.21.15.151:6802/2666240674","state":["exists","up"]},{"osd":5,"uuid":"4c72b1d7-28f6-4ee3-bb80-57ff924abc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6804","nonce":2718612978}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6805","nonce":2718612978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6807","nonce":2718612978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6806","nonce":2718612978}]},"public_addr":"172.21.15.151:6804/2718612978","cluster_addr":"172.21.15.151:6805/2718612978","heartbeat_back_addr":"172.21.15.151:6807/2718612978","heartbeat_front_addr":"172.21.15.151:6806/2718612978","state":["exists","up"]},{"osd":6,"uuid":"d852205d-6b9d-4a4f-83b4-a0e59aa37657","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6808","nonce":1497503104}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6809","nonce":1497503104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6811","nonce":1497503104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6810","nonce":1497503104}]},"public_addr":"172.21.15.151:6808/1497503104","cluster_addr":"172.21.15.151:6809/1497503104","heartbeat_back_addr":"172.21.15.151:6811/1497503104","heartbeat_front_addr":"172.21.15.151:6810/1497503104","state":["exists","up"]},{"osd":7,"uuid":"cd9fca37-a6f2-46fc-90de-cd3790ad3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6812","nonce":683127336}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6813","nonce":683127336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6815","nonce":683127336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6814","nonce":683127336}]},"public_addr":"172.21.15.151:6812/683127336","cluster_addr":"172.21.15.151:6813/683127336","heartbeat_back_addr":"172.21.15.151:6815/683127336","heartbeat_front_addr":"172.21.15.151:6814/683127336","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:10:26.284321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:10:52.628310+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:11:18.278275+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:11:44.195150+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:12:07.431888+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:12:32.407393+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:12:57.133251+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:13:23.236080+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/2796836265":"2024-03-13T03:09:02.315045+0000","172.21.15.94:0/1664309417":"2024-03-13T03:08:34.850802+0000","172.21.15.94:0/3802544616":"2024-03-13T03:08:34.850802+0000","172.21.15.94:6800/3548280253":"2024-03-13T03:09:02.315045+0000","172.21.15.94:6800/2955156665":"2024-03-13T03:08:34.850802+0000","172.21.15.94:0/782403058":"2024-03-13T03:08:34.850802+0000","172.21.15.94:0/3132181":"2024-03-13T03:09:02.315045+0000","172.21.15.94:0/3815053600":"2024-03-13T03:09:02.315045+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:13:28.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.465+0000 7f3fbb03c700 1 -- 172.21.15.94:0/2287402302 >> v1:172.21.15.94:6800/1299282424 conn(0x7f3fa406ea10 legacy=0x7f3fa4070ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:28.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.465+0000 7f3fbb03c700 1 -- 172.21.15.94:0/2287402302 >> v1:172.21.15.94:6789/0 conn(0x7f3fb410cc80 legacy=0x7f3fb41a7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:28.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.465+0000 7f3fbb03c700 1 -- 172.21.15.94:0/2287402302 shutdown_connections 2024-03-12T03:13:28.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.465+0000 7f3fbb03c700 1 -- 172.21.15.94:0/2287402302 >> 172.21.15.94:0/2287402302 conn(0x7f3fb4076ec0 msgr2=0x7f3fb4108870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:28.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.465+0000 7f3fbb03c700 1 -- 172.21.15.94:0/2287402302 shutdown_connections 2024-03-12T03:13:28.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:28.465+0000 7f3fbb03c700 1 -- 172.21.15.94:0/2287402302 wait complete. 2024-03-12T03:13:28.853 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:28 smithi151 ceph-mon[32923]: Detected new or changed devices on smithi151 2024-03-12T03:13:28.853 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:28 smithi151 ceph-mon[32923]: Adjusting osd_memory_target on smithi151 to 4272M 2024-03-12T03:13:28.853 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:28 smithi151 ceph-mon[32923]: osdmap e57: 8 total, 8 up, 8 in 2024-03-12T03:13:28.853 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2287402302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:13:28.910 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:28 smithi094 ceph-mon[29291]: Detected new or changed devices on smithi151 2024-03-12T03:13:28.910 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:28 smithi094 ceph-mon[29291]: Adjusting osd_memory_target on smithi151 to 4272M 2024-03-12T03:13:28.910 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:28 smithi094 ceph-mon[29291]: osdmap e57: 8 total, 8 up, 8 in 2024-03-12T03:13:28.910 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2287402302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:13:28.910 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:28 smithi094 ceph-mon[40484]: Detected new or changed devices on smithi151 2024-03-12T03:13:28.910 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:28 smithi094 ceph-mon[40484]: Adjusting osd_memory_target on smithi151 to 4272M 2024-03-12T03:13:28.911 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:28 smithi094 ceph-mon[40484]: osdmap e57: 8 total, 8 up, 8 in 2024-03-12T03:13:28.911 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2287402302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:13:29.131 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T03:11:20.645672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T03:13:29.131 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-12T03:13:29.373 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:13:29.809 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:29 smithi094 ceph-mon[29291]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:13:29.810 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:29 smithi094 ceph-mon[29291]: osdmap e58: 8 total, 8 up, 8 in 2024-03-12T03:13:29.810 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:29 smithi094 ceph-mon[40484]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:13:29.810 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:29 smithi094 ceph-mon[40484]: osdmap e58: 8 total, 8 up, 8 in 2024-03-12T03:13:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:29 smithi151 ceph-mon[32923]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:13:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:29 smithi151 ceph-mon[32923]: osdmap e58: 8 total, 8 up, 8 in 2024-03-12T03:13:31.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.017+0000 7f7d66250700 1 -- 172.21.15.94:0/989157339 >> v1:172.21.15.94:6789/0 conn(0x7f7d6010d500 legacy=0x7f7d6010f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:31.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.017+0000 7f7d66250700 1 -- 172.21.15.94:0/989157339 shutdown_connections 2024-03-12T03:13:31.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.017+0000 7f7d66250700 1 -- 172.21.15.94:0/989157339 >> 172.21.15.94:0/989157339 conn(0x7f7d600fc570 msgr2=0x7f7d600fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:31.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.017+0000 7f7d66250700 1 -- 172.21.15.94:0/989157339 shutdown_connections 2024-03-12T03:13:31.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.017+0000 7f7d66250700 1 -- 172.21.15.94:0/989157339 wait complete. 2024-03-12T03:13:31.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.017+0000 7f7d66250700 1 Processor -- start 2024-03-12T03:13:31.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.018+0000 7f7d66250700 1 -- start start 2024-03-12T03:13:31.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.018+0000 7f7d66250700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7d601a8e80 con 0x7f7d60109930 2024-03-12T03:13:31.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.018+0000 7f7d66250700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7d601aa0a0 con 0x7f7d6010d500 2024-03-12T03:13:31.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.018+0000 7f7d66250700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7d601ab2e0 con 0x7f7d60106a80 2024-03-12T03:13:31.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.018+0000 7f7d5f7fe700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f7d60106a80 0x7f7d600fe440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:36128/0 (socket says 172.21.15.94:36128) 2024-03-12T03:13:31.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.018+0000 7f7d5f7fe700 1 -- 172.21.15.94:0/3226802217 learned_addr learned my addr 172.21.15.94:0/3226802217 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:13:31.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.018+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880390125 0 0) 0x7f7d601ab2e0 con 0x7f7d60106a80 2024-03-12T03:13:31.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.018+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d480035b0 con 0x7f7d60106a80 2024-03-12T03:13:31.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.018+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274161776 0 0) 0x7f7d601a8e80 con 0x7f7d60109930 2024-03-12T03:13:31.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.018+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d601ab2e0 con 0x7f7d60109930 2024-03-12T03:13:31.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.018+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1966331374 0 0) 0x7f7d601ab2e0 con 0x7f7d60109930 2024-03-12T03:13:31.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.018+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d601a8e80 con 0x7f7d60109930 2024-03-12T03:13:31.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.018+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f7d54001830 con 0x7f7d60109930 2024-03-12T03:13:31.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.019+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621569995 0 0) 0x7f7d601aa0a0 con 0x7f7d6010d500 2024-03-12T03:13:31.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.019+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d601ab2e0 con 0x7f7d6010d500 2024-03-12T03:13:31.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.019+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2995759121 0 0) 0x7f7d480035b0 con 0x7f7d60106a80 2024-03-12T03:13:31.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.019+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d601aa0a0 con 0x7f7d60106a80 2024-03-12T03:13:31.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.019+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f7d50001890 con 0x7f7d60106a80 2024-03-12T03:13:31.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.019+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2994192679 0 0) 0x7f7d601a8e80 con 0x7f7d60109930 2024-03-12T03:13:31.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.019+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 >> v1:172.21.15.94:6790/0 conn(0x7f7d60106a80 legacy=0x7f7d600fe440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:31.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.019+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 >> v1:172.21.15.151:6789/0 conn(0x7f7d6010d500 legacy=0x7f7d601a7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:31.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.019+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d601ac520 con 0x7f7d60109930 2024-03-12T03:13:31.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.019+0000 7f7d66250700 1 -- 172.21.15.94:0/3226802217 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d601a90b0 con 0x7f7d60109930 2024-03-12T03:13:31.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.019+0000 7f7d66250700 1 -- 172.21.15.94:0/3226802217 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d601a9570 con 0x7f7d60109930 2024-03-12T03:13:31.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.019+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f7d54003b30 con 0x7f7d60109930 2024-03-12T03:13:31.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.019+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f7d54004a10 con 0x7f7d60109930 2024-03-12T03:13:31.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.020+0000 7f7d66250700 1 -- 172.21.15.94:0/3226802217 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c005120 con 0x7f7d60109930 2024-03-12T03:13:31.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.020+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 15) v1 ==== 91719+0+0 (unknown 3315827787 0 0) 0x7f7d54003680 con 0x7f7d60109930 2024-03-12T03:13:31.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.021+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (unknown 126689524 0 0) 0x7f7d540876e0 con 0x7f7d60109930 2024-03-12T03:13:31.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.025+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f7d54055800 con 0x7f7d60109930 2024-03-12T03:13:31.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.217+0000 7f7d66250700 1 -- 172.21.15.94:0/3226802217 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d4c005c20 con 0x7f7d60109930 2024-03-12T03:13:31.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.218+0000 7f7d5cff9700 1 -- 172.21.15.94:0/3226802217 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (unknown 3844627101 0 2170607528) 0x7f7d54055100 con 0x7f7d60109930 2024-03-12T03:13:31.221 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-12T03:13:31.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.220+0000 7f7d66250700 1 -- 172.21.15.94:0/3226802217 >> v1:172.21.15.94:6800/1299282424 conn(0x7f7d4806ec00 legacy=0x7f7d480710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:31.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.220+0000 7f7d66250700 1 -- 172.21.15.94:0/3226802217 >> v1:172.21.15.94:6789/0 conn(0x7f7d60109930 legacy=0x7f7d601a3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:31.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.220+0000 7f7d66250700 1 -- 172.21.15.94:0/3226802217 shutdown_connections 2024-03-12T03:13:31.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.220+0000 7f7d66250700 1 -- 172.21.15.94:0/3226802217 >> 172.21.15.94:0/3226802217 conn(0x7f7d600fc570 msgr2=0x7f7d600ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:31.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.220+0000 7f7d66250700 1 -- 172.21.15.94:0/3226802217 shutdown_connections 2024-03-12T03:13:31.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:13:31.220+0000 7f7d66250700 1 -- 172.21.15.94:0/3226802217 wait complete. 2024-03-12T03:13:31.813 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:31 smithi094 ceph-mon[29291]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:13:31.813 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3226802217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:13:31.813 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:31 smithi094 ceph-mon[40484]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:13:31.813 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3226802217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:13:31.839 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi094 2024-03-12T03:13:31.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi094=foo.a' 2024-03-12T03:13:31.870 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:31 smithi151 ceph-mon[32923]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:13:31.870 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3226802217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:13:32.092 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:13:33.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.676+0000 7f3d795dc700 1 -- 172.21.15.151:0/2619019111 >> v1:172.21.15.151:6789/0 conn(0x7f3d74100ae0 legacy=0x7f3d74100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:33.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.676+0000 7f3d795dc700 1 -- 172.21.15.151:0/2619019111 shutdown_connections 2024-03-12T03:13:33.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.676+0000 7f3d795dc700 1 -- 172.21.15.151:0/2619019111 >> 172.21.15.151:0/2619019111 conn(0x7f3d740fc0a0 msgr2=0x7f3d740fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:33.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.676+0000 7f3d795dc700 1 -- 172.21.15.151:0/2619019111 shutdown_connections 2024-03-12T03:13:33.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.676+0000 7f3d795dc700 1 -- 172.21.15.151:0/2619019111 wait complete. 2024-03-12T03:13:33.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.677+0000 7f3d795dc700 1 Processor -- start 2024-03-12T03:13:33.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.677+0000 7f3d795dc700 1 -- start start 2024-03-12T03:13:33.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.677+0000 7f3d795dc700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3d741967d0 con 0x7f3d74108c50 2024-03-12T03:13:33.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.677+0000 7f3d795dc700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3d741a1fc0 con 0x7f3d74104f10 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.677+0000 7f3d795dc700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3d741a31a0 con 0x7f3d74100ae0 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d727fc700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f3d74104f10 0x7f3d7419c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:59108/0 (socket says 172.21.15.151:59108) 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d727fc700 1 -- 172.21.15.151:0/1483683244 learned_addr learned my addr 172.21.15.151:0/1483683244 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214300537 0 0) 0x7f3d741a1fc0 con 0x7f3d74104f10 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d5c0035b0 con 0x7f3d74104f10 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 500584288 0 0) 0x7f3d5c0035b0 con 0x7f3d74104f10 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d741a1fc0 con 0x7f3d74104f10 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f3d68004090 con 0x7f3d74104f10 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245342669 0 0) 0x7f3d741967d0 con 0x7f3d74108c50 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d5c0035b0 con 0x7f3d74108c50 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682147381 0 0) 0x7f3d741a31a0 con 0x7f3d74100ae0 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d741967d0 con 0x7f3d74100ae0 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2074566171 0 0) 0x7f3d741a1fc0 con 0x7f3d74104f10 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 >> v1:172.21.15.94:6790/0 conn(0x7f3d74100ae0 legacy=0x7f3d74195c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 >> v1:172.21.15.94:6789/0 conn(0x7f3d74108c50 legacy=0x7f3d7419f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:33.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d741a43c0 con 0x7f3d74104f10 2024-03-12T03:13:33.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d795dc700 1 -- 172.21.15.151:0/1483683244 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d741a1010 con 0x7f3d74104f10 2024-03-12T03:13:33.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.678+0000 7f3d795dc700 1 -- 172.21.15.151:0/1483683244 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d741a14d0 con 0x7f3d74104f10 2024-03-12T03:13:33.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.679+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f3d680013e0 con 0x7f3d74104f10 2024-03-12T03:13:33.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.679+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f3d68005ce0 con 0x7f3d74104f10 2024-03-12T03:13:33.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.679+0000 7f3d795dc700 1 -- 172.21.15.151:0/1483683244 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d60005150 con 0x7f3d74104f10 2024-03-12T03:13:33.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.680+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 15) v1 ==== 91719+0+0 (unknown 3315827787 0 0) 0x7f3d68000f50 con 0x7f3d74104f10 2024-03-12T03:13:33.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.680+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (unknown 126689524 0 0) 0x7f3d680888d0 con 0x7f3d74104f10 2024-03-12T03:13:33.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.685+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f3d68056ad0 con 0x7f3d74104f10 2024-03-12T03:13:33.687 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:33 smithi151 ceph-mon[32923]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:13:33.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.887+0000 7f3d795dc700 1 -- 172.21.15.151:0/1483683244 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi094=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d60000bc0 con 0x7f3d5c06ec20 2024-03-12T03:13:33.897 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.894+0000 7f3d5bfff700 1 -- 172.21.15.151:0/1483683244 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (unknown 0 0 1123153589) 0x7f3d60000bc0 con 0x7f3d5c06ec20 2024-03-12T03:13:33.897 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled rgw.foo.a update... 2024-03-12T03:13:33.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.897+0000 7f3d795dc700 1 -- 172.21.15.151:0/1483683244 >> v1:172.21.15.94:6800/1299282424 conn(0x7f3d5c06ec20 legacy=0x7f3d5c0710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:33.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.897+0000 7f3d795dc700 1 -- 172.21.15.151:0/1483683244 >> v1:172.21.15.151:6789/0 conn(0x7f3d74104f10 legacy=0x7f3d7419c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:33.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.897+0000 7f3d795dc700 1 -- 172.21.15.151:0/1483683244 shutdown_connections 2024-03-12T03:13:33.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.897+0000 7f3d795dc700 1 -- 172.21.15.151:0/1483683244 >> 172.21.15.151:0/1483683244 conn(0x7f3d740fc0a0 msgr2=0x7f3d741082f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:33.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.897+0000 7f3d795dc700 1 -- 172.21.15.151:0/1483683244 shutdown_connections 2024-03-12T03:13:33.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:33.897+0000 7f3d795dc700 1 -- 172.21.15.151:0/1483683244 wait complete. 2024-03-12T03:13:33.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:33 smithi094 ceph-mon[29291]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:13:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:33 smithi094 ceph-mon[40484]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:13:34.648 DEBUG:teuthology.orchestra.run.smithi094:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-12T03:13:34.651 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi151 2024-03-12T03:13:34.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-12T03:13:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:13:34.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[29291]: from='client.24355 v1:172.21.15.151:0/1483683244' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi094=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[29291]: Saving service rgw.foo.a spec with placement smithi094=foo.a;count:1 2024-03-12T03:13:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[29291]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-12T03:13:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:13:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:13:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[40484]: from='client.24355 v1:172.21.15.151:0/1483683244' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi094=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[40484]: Saving service rgw.foo.a spec with placement smithi094=foo.a;count:1 2024-03-12T03:13:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[40484]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-12T03:13:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:13:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:13:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:34.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:34 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:35.154 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:34 smithi151 ceph-mon[32923]: from='client.24355 v1:172.21.15.151:0/1483683244' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi094=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:35.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:34 smithi151 ceph-mon[32923]: Saving service rgw.foo.a spec with placement smithi094=foo.a;count:1 2024-03-12T03:13:35.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:35.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:35.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:34 smithi151 ceph-mon[32923]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-12T03:13:35.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:35.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:35.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:35.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:13:35.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:13:35.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:35.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:34 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:36.143 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:35 smithi151 ceph-mon[32923]: Deploying daemon rgw.foo.a on smithi094 2024-03-12T03:13:36.196 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:35 smithi094 ceph-mon[40484]: Deploying daemon rgw.foo.a on smithi094 2024-03-12T03:13:36.197 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:35 smithi094 ceph-mon[29291]: Deploying daemon rgw.foo.a on smithi094 2024-03-12T03:13:36.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.366+0000 7fcfa73db700 1 -- 172.21.15.151:0/428094488 >> v1:172.21.15.151:6789/0 conn(0x7fcfa0109d70 legacy=0x7fcfa010c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:36.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.366+0000 7fcfa73db700 1 -- 172.21.15.151:0/428094488 shutdown_connections 2024-03-12T03:13:36.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.366+0000 7fcfa73db700 1 -- 172.21.15.151:0/428094488 >> 172.21.15.151:0/428094488 conn(0x7fcfa00fd1e0 msgr2=0x7fcfa00ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:36.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.366+0000 7fcfa73db700 1 -- 172.21.15.151:0/428094488 shutdown_connections 2024-03-12T03:13:36.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.366+0000 7fcfa73db700 1 -- 172.21.15.151:0/428094488 wait complete. 2024-03-12T03:13:36.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.367+0000 7fcfa73db700 1 Processor -- start 2024-03-12T03:13:36.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.367+0000 7fcfa73db700 1 -- start start 2024-03-12T03:13:36.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.367+0000 7fcfa73db700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcfa01a4480 con 0x7fcfa0109d70 2024-03-12T03:13:36.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.367+0000 7fcfa73db700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcfa01a56c0 con 0x7fcfa0106030 2024-03-12T03:13:36.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.367+0000 7fcfa73db700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcfa01a6900 con 0x7fcfa0101c00 2024-03-12T03:13:36.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.367+0000 7fcfa4976700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fcfa0106030 0x7fcfa019f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:57822/0 (socket says 172.21.15.151:57822) 2024-03-12T03:13:36.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.367+0000 7fcfa4976700 1 -- 172.21.15.151:0/3821559818 learned_addr learned my addr 172.21.15.151:0/3821559818 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:13:36.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.367+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015754302 0 0) 0x7fcfa01a56c0 con 0x7fcfa0106030 2024-03-12T03:13:36.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.367+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf8c0035b0 con 0x7fcfa0106030 2024-03-12T03:13:36.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.367+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224669546 0 0) 0x7fcfa01a4480 con 0x7fcfa0109d70 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.367+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa01a56c0 con 0x7fcfa0109d70 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.367+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225679219 0 0) 0x7fcfa01a6900 con 0x7fcfa0101c00 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.367+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa01a4480 con 0x7fcfa0101c00 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 427687968 0 0) 0x7fcf8c0035b0 con 0x7fcfa0106030 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfa01a6900 con 0x7fcfa0106030 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fcf90001910 con 0x7fcfa0106030 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1063695999 0 0) 0x7fcfa01a56c0 con 0x7fcfa0109d70 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf8c0035b0 con 0x7fcfa0109d70 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fcf98001c80 con 0x7fcfa0109d70 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3843082636 0 0) 0x7fcfa01a4480 con 0x7fcfa0101c00 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfa01a56c0 con 0x7fcfa0101c00 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fcf9c001830 con 0x7fcfa0101c00 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3836375284 0 0) 0x7fcfa01a6900 con 0x7fcfa0106030 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 >> v1:172.21.15.94:6790/0 conn(0x7fcfa0101c00 legacy=0x7fcfa0075aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 >> v1:172.21.15.94:6789/0 conn(0x7fcfa0109d70 legacy=0x7fcfa01a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:36.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa01a7b40 con 0x7fcfa0106030 2024-03-12T03:13:36.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcfa73db700 1 -- 172.21.15.151:0/3821559818 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa01a46b0 con 0x7fcfa0106030 2024-03-12T03:13:36.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcfa73db700 1 -- 172.21.15.151:0/3821559818 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa01a4bf0 con 0x7fcfa0106030 2024-03-12T03:13:36.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fcf90001d00 con 0x7fcfa0106030 2024-03-12T03:13:36.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.368+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fcf90005d70 con 0x7fcfa0106030 2024-03-12T03:13:36.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.369+0000 7fcfa73db700 1 -- 172.21.15.151:0/3821559818 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf84005150 con 0x7fcfa0106030 2024-03-12T03:13:36.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.369+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 15) v1 ==== 91719+0+0 (unknown 3315827787 0 0) 0x7fcf90003950 con 0x7fcfa0106030 2024-03-12T03:13:36.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.370+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (unknown 126689524 0 0) 0x7fcf900887f0 con 0x7fcfa0106030 2024-03-12T03:13:36.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.374+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fcf900568b0 con 0x7fcfa0106030 2024-03-12T03:13:36.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.566+0000 7fcfa73db700 1 -- 172.21.15.151:0/3821559818 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7fcf84005440 con 0x7fcfa0106030 2024-03-12T03:13:36.876 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:13:36 smithi094 systemd[1]: Starting Ceph rgw.foo.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:13:36.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.910+0000 7fcf967fc700 1 -- 172.21.15.151:0/3821559818 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v59) v1 ==== 160+0+0 (unknown 1751278495 0 0) 0x7fcf900561b0 con 0x7fcfa0106030 2024-03-12T03:13:36.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.912+0000 7fcfa73db700 1 -- 172.21.15.151:0/3821559818 >> v1:172.21.15.94:6800/1299282424 conn(0x7fcf8c06ec10 legacy=0x7fcf8c0710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:36.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.912+0000 7fcfa73db700 1 -- 172.21.15.151:0/3821559818 >> v1:172.21.15.151:6789/0 conn(0x7fcfa0106030 legacy=0x7fcfa019f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:36.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.912+0000 7fcfa73db700 1 -- 172.21.15.151:0/3821559818 shutdown_connections 2024-03-12T03:13:36.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.912+0000 7fcfa73db700 1 -- 172.21.15.151:0/3821559818 >> 172.21.15.151:0/3821559818 conn(0x7fcfa00fd1e0 msgr2=0x7fcfa0108c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:36.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.912+0000 7fcfa73db700 1 -- 172.21.15.151:0/3821559818 shutdown_connections 2024-03-12T03:13:36.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:36.912+0000 7fcfa73db700 1 -- 172.21.15.151:0/3821559818 wait complete. 2024-03-12T03:13:36.916 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'datapool' created 2024-03-12T03:13:37.186 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:36 smithi094 ceph-mon[29291]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-12T03:13:37.186 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/3821559818' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-12T03:13:37.186 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:36 smithi094 ceph-mon[29291]: from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-12T03:13:37.186 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:36 smithi094 ceph-mon[40484]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-12T03:13:37.186 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/3821559818' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-12T03:13:37.186 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:36 smithi094 ceph-mon[40484]: from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-12T03:13:37.190 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:13:36 smithi094 podman[67566]: 2024-03-12 03:13:36.776110951 +0000 UTC m=+0.017454887 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:13:37.190 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:13:36 smithi094 podman[67566]: 2024-03-12 03:13:36.905848736 +0000 UTC m=+0.147192656 container create 6b571fe46f3e20ba9e07b75adbe95e9d466797351a6fcf0e4f490e0890cdb618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-rgw-foo-a, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-12T03:13:37.190 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:13:37 smithi094 podman[67566]: 2024-03-12 03:13:37.184255759 +0000 UTC m=+0.425599673 container init 6b571fe46f3e20ba9e07b75adbe95e9d466797351a6fcf0e4f490e0890cdb618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-rgw-foo-a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-12T03:13:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:36 smithi151 ceph-mon[32923]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-12T03:13:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/3821559818' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-12T03:13:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:36 smithi151 ceph-mon[32923]: from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-12T03:13:37.459 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:13:37 smithi094 podman[67566]: 2024-03-12 03:13:37.187011718 +0000 UTC m=+0.428355640 container start 6b571fe46f3e20ba9e07b75adbe95e9d466797351a6fcf0e4f490e0890cdb618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:13:37.460 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:13:37 smithi094 bash[67566]: 6b571fe46f3e20ba9e07b75adbe95e9d466797351a6fcf0e4f490e0890cdb618 2024-03-12T03:13:37.460 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:13:37 smithi094 systemd[1]: Started Ceph rgw.foo.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:13:37.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- rbd pool init datapool 2024-03-12T03:13:37.828 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:13:38.170 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:37 smithi151 ceph-mon[32923]: from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-12T03:13:38.170 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:37 smithi151 ceph-mon[32923]: osdmap e59: 8 total, 8 up, 8 in 2024-03-12T03:13:38.170 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.170 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.170 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.170 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:37 smithi151 ceph-mon[32923]: Saving service rgw.foo.a spec with placement smithi094=foo.a;count:1 2024-03-12T03:13:38.170 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.170 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.171 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:37 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[29291]: from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-12T03:13:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[29291]: osdmap e59: 8 total, 8 up, 8 in 2024-03-12T03:13:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[29291]: Saving service rgw.foo.a spec with placement smithi094=foo.a;count:1 2024-03-12T03:13:38.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:38.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[40484]: from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-12T03:13:38.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[40484]: osdmap e59: 8 total, 8 up, 8 in 2024-03-12T03:13:38.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[40484]: Saving service rgw.foo.a spec with placement smithi094=foo.a;count:1 2024-03-12T03:13:38.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:38.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:37 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:39.185 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:38 smithi151 ceph-mon[32923]: osdmap e60: 8 total, 8 up, 8 in 2024-03-12T03:13:39.186 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1887210402' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:13:39.186 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:38 smithi151 ceph-mon[32923]: from='client.24370 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:13:39.186 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:38 smithi151 ceph-mon[32923]: pgmap v188: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:13:39.186 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:38 smithi151 ceph-mon[32923]: from='client.24370 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T03:13:39.186 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:38 smithi151 ceph-mon[32923]: osdmap e61: 8 total, 8 up, 8 in 2024-03-12T03:13:39.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:38 smithi094 ceph-mon[29291]: osdmap e60: 8 total, 8 up, 8 in 2024-03-12T03:13:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1887210402' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:13:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:38 smithi094 ceph-mon[29291]: from='client.24370 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:13:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:38 smithi094 ceph-mon[29291]: pgmap v188: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:13:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:38 smithi094 ceph-mon[29291]: from='client.24370 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T03:13:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:38 smithi094 ceph-mon[29291]: osdmap e61: 8 total, 8 up, 8 in 2024-03-12T03:13:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:38 smithi094 ceph-mon[40484]: osdmap e60: 8 total, 8 up, 8 in 2024-03-12T03:13:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1887210402' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:13:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:38 smithi094 ceph-mon[40484]: from='client.24370 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:13:39.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:38 smithi094 ceph-mon[40484]: pgmap v188: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:13:39.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:38 smithi094 ceph-mon[40484]: from='client.24370 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T03:13:39.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:38 smithi094 ceph-mon[40484]: osdmap e61: 8 total, 8 up, 8 in 2024-03-12T03:13:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[29291]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:13:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:40.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[29291]: Checking dashboard <-> RGW credentials 2024-03-12T03:13:40.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/817530945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-12T03:13:40.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[29291]: from='client.24397 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-12T03:13:40.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[29291]: from='client.24397 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-12T03:13:40.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[29291]: osdmap e62: 8 total, 8 up, 8 in 2024-03-12T03:13:40.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:13:40.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[40484]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:13:40.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:40.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:40.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:40.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:40.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[40484]: Checking dashboard <-> RGW credentials 2024-03-12T03:13:40.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/817530945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-12T03:13:40.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[40484]: from='client.24397 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-12T03:13:40.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[40484]: from='client.24397 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-12T03:13:40.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[40484]: osdmap e62: 8 total, 8 up, 8 in 2024-03-12T03:13:40.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:13:40.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:39 smithi151 ceph-mon[32923]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:13:40.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:40.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:40.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:40.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:39 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:40.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:39 smithi151 ceph-mon[32923]: Checking dashboard <-> RGW credentials 2024-03-12T03:13:40.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/817530945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-12T03:13:40.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:39 smithi151 ceph-mon[32923]: from='client.24397 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-12T03:13:40.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:39 smithi151 ceph-mon[32923]: from='client.24397 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-12T03:13:40.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:39 smithi151 ceph-mon[32923]: osdmap e62: 8 total, 8 up, 8 in 2024-03-12T03:13:40.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:13:41.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:13:41.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:40 smithi094 ceph-mon[29291]: pgmap v191: 68 pgs: 6 creating+peering, 13 active+clean, 49 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:13:41.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:13:41.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:13:41.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:40 smithi094 ceph-mon[29291]: osdmap e63: 8 total, 8 up, 8 in 2024-03-12T03:13:41.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:13:41.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:40 smithi094 ceph-mon[40484]: pgmap v191: 68 pgs: 6 creating+peering, 13 active+clean, 49 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:13:41.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:13:41.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:13:41.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:40 smithi094 ceph-mon[40484]: osdmap e63: 8 total, 8 up, 8 in 2024-03-12T03:13:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:13:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:40 smithi151 ceph-mon[32923]: pgmap v191: 68 pgs: 6 creating+peering, 13 active+clean, 49 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:13:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:13:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:13:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:40 smithi151 ceph-mon[32923]: osdmap e63: 8 total, 8 up, 8 in 2024-03-12T03:13:42.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.151 --placement '1;smithi151=iscsi.a' 2024-03-12T03:13:42.732 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:13:43.167 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:42 smithi151 ceph-mon[32923]: osdmap e64: 8 total, 8 up, 8 in 2024-03-12T03:13:43.168 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:13:43.168 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:13:43.168 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:42 smithi151 ceph-mon[32923]: pgmap v194: 100 pgs: 11 creating+peering, 38 active+clean, 51 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-12T03:13:43.168 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:42 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:43.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:42 smithi094 ceph-mon[40484]: osdmap e64: 8 total, 8 up, 8 in 2024-03-12T03:13:43.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:13:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:13:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:42 smithi094 ceph-mon[40484]: pgmap v194: 100 pgs: 11 creating+peering, 38 active+clean, 51 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-12T03:13:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:42 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:42 smithi094 ceph-mon[29291]: osdmap e64: 8 total, 8 up, 8 in 2024-03-12T03:13:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:13:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:13:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:42 smithi094 ceph-mon[29291]: pgmap v194: 100 pgs: 11 creating+peering, 38 active+clean, 51 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-12T03:13:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:42 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:44.032 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:13:44.033 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:13:44.033 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:43 smithi151 ceph-mon[32923]: osdmap e65: 8 total, 8 up, 8 in 2024-03-12T03:13:44.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:13:44.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:13:44.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:43 smithi094 ceph-mon[40484]: osdmap e65: 8 total, 8 up, 8 in 2024-03-12T03:13:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:13:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:13:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:43 smithi094 ceph-mon[29291]: osdmap e65: 8 total, 8 up, 8 in 2024-03-12T03:13:44.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.215+0000 7fd951ee5700 1 -- 172.21.15.151:0/3495931392 >> v1:172.21.15.151:6789/0 conn(0x7fd94c104ee0 legacy=0x7fd94c107330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:44.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.215+0000 7fd951ee5700 1 -- 172.21.15.151:0/3495931392 shutdown_connections 2024-03-12T03:13:44.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.215+0000 7fd951ee5700 1 -- 172.21.15.151:0/3495931392 >> 172.21.15.151:0/3495931392 conn(0x7fd94c0fc070 msgr2=0x7fd94c0fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:44.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.215+0000 7fd951ee5700 1 -- 172.21.15.151:0/3495931392 shutdown_connections 2024-03-12T03:13:44.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.215+0000 7fd951ee5700 1 -- 172.21.15.151:0/3495931392 wait complete. 2024-03-12T03:13:44.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.216+0000 7fd951ee5700 1 Processor -- start 2024-03-12T03:13:44.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.216+0000 7fd951ee5700 1 -- start start 2024-03-12T03:13:44.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.216+0000 7fd951ee5700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd94c196c10 con 0x7fd94c108c20 2024-03-12T03:13:44.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.216+0000 7fd951ee5700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd94c1a2400 con 0x7fd94c104ee0 2024-03-12T03:13:44.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.216+0000 7fd951ee5700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd94c1a35e0 con 0x7fd94c100ab0 2024-03-12T03:13:44.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.216+0000 7fd942ffd700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd94c104ee0 0x7fd94c19c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:57844/0 (socket says 172.21.15.151:57844) 2024-03-12T03:13:44.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.216+0000 7fd942ffd700 1 -- 172.21.15.151:0/3868626772 learned_addr learned my addr 172.21.15.151:0/3868626772 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:13:44.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.216+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358034703 0 0) 0x7fd94c1a2400 con 0x7fd94c104ee0 2024-03-12T03:13:44.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.216+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9380035b0 con 0x7fd94c104ee0 2024-03-12T03:13:44.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.216+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260488403 0 0) 0x7fd94c196c10 con 0x7fd94c108c20 2024-03-12T03:13:44.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.216+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd94c1a2400 con 0x7fd94c108c20 2024-03-12T03:13:44.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.216+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18855748 0 0) 0x7fd94c1a35e0 con 0x7fd94c100ab0 2024-03-12T03:13:44.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.216+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd94c196c10 con 0x7fd94c100ab0 2024-03-12T03:13:44.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.217+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3931122157 0 0) 0x7fd9380035b0 con 0x7fd94c104ee0 2024-03-12T03:13:44.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.217+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd94c1a35e0 con 0x7fd94c104ee0 2024-03-12T03:13:44.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.217+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fd934001430 con 0x7fd94c104ee0 2024-03-12T03:13:44.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.217+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 519854871 0 0) 0x7fd94c1a35e0 con 0x7fd94c104ee0 2024-03-12T03:13:44.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.217+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 >> v1:172.21.15.94:6790/0 conn(0x7fd94c100ab0 legacy=0x7fd94c1960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:44.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.217+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 >> v1:172.21.15.94:6789/0 conn(0x7fd94c108c20 legacy=0x7fd94c19fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:44.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.217+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94c1a4800 con 0x7fd94c104ee0 2024-03-12T03:13:44.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.217+0000 7fd951ee5700 1 -- 172.21.15.151:0/3868626772 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd94c1a1450 con 0x7fd94c104ee0 2024-03-12T03:13:44.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.217+0000 7fd951ee5700 1 -- 172.21.15.151:0/3868626772 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd94c1a1990 con 0x7fd94c104ee0 2024-03-12T03:13:44.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.217+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fd9340011a0 con 0x7fd94c104ee0 2024-03-12T03:13:44.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.217+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fd934004ec0 con 0x7fd94c104ee0 2024-03-12T03:13:44.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.218+0000 7fd951ee5700 1 -- 172.21.15.151:0/3868626772 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd92c005150 con 0x7fd94c104ee0 2024-03-12T03:13:44.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.218+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 15) v1 ==== 91719+0+0 (unknown 3315827787 0 0) 0x7fd934005080 con 0x7fd94c104ee0 2024-03-12T03:13:44.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.219+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(66..66 src has 1..66) v4 ==== 5915+0+0 (unknown 2102073363 0 0) 0x7fd934089170 con 0x7fd94c104ee0 2024-03-12T03:13:44.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.223+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fd934056b50 con 0x7fd94c104ee0 2024-03-12T03:13:44.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.431+0000 7fd951ee5700 1 -- 172.21.15.151:0/3868626772 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.151", "placement": "1;smithi151=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7fd92c000c90 con 0x7fd93806ec20 2024-03-12T03:13:44.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.439+0000 7fd9497fa700 1 -- 172.21.15.151:0/3868626772 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (unknown 0 0 803663096) 0x7fd92c000c90 con 0x7fd93806ec20 2024-03-12T03:13:44.441 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled iscsi.datapool update... 2024-03-12T03:13:44.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.441+0000 7fd951ee5700 1 -- 172.21.15.151:0/3868626772 >> v1:172.21.15.94:6800/1299282424 conn(0x7fd93806ec20 legacy=0x7fd9380710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:44.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.441+0000 7fd951ee5700 1 -- 172.21.15.151:0/3868626772 >> v1:172.21.15.151:6789/0 conn(0x7fd94c104ee0 legacy=0x7fd94c19c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:44.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.441+0000 7fd951ee5700 1 -- 172.21.15.151:0/3868626772 shutdown_connections 2024-03-12T03:13:44.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.441+0000 7fd951ee5700 1 -- 172.21.15.151:0/3868626772 >> 172.21.15.151:0/3868626772 conn(0x7fd94c0fc070 msgr2=0x7fd94c107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:44.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.441+0000 7fd951ee5700 1 -- 172.21.15.151:0/3868626772 shutdown_connections 2024-03-12T03:13:44.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:44.441+0000 7fd951ee5700 1 -- 172.21.15.151:0/3868626772 wait complete. 2024-03-12T03:13:45.020 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:44 smithi151 ceph-mon[32923]: osdmap e66: 8 total, 8 up, 8 in 2024-03-12T03:13:45.020 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:13:45.020 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:13:45.020 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:44 smithi151 ceph-mon[32923]: pgmap v197: 132 pgs: 14 creating+peering, 72 active+clean, 46 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-12T03:13:45.020 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:44 smithi151 ceph-mon[32923]: from='client.24418 v1:172.21.15.151:0/3868626772' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.151", "placement": "1;smithi151=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:45.021 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:44 smithi151 ceph-mon[32923]: Saving service iscsi.datapool spec with placement smithi151=iscsi.a;count:1 2024-03-12T03:13:45.021 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:44 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:45.048 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-12T03:13:45.049 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:13:45.049 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-12T03:13:45.084 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-12T03:13:45.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-12T03:13:45.114 DEBUG:teuthology.orchestra.run.smithi151:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@iscsi.iscsi.a.service 2024-03-12T03:13:45.158 INFO:tasks.cephadm:Adding prometheus.a on smithi151 2024-03-12T03:13:45.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi151=a' 2024-03-12T03:13:45.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[29291]: osdmap e66: 8 total, 8 up, 8 in 2024-03-12T03:13:45.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:13:45.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:13:45.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[29291]: pgmap v197: 132 pgs: 14 creating+peering, 72 active+clean, 46 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-12T03:13:45.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[29291]: from='client.24418 v1:172.21.15.151:0/3868626772' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.151", "placement": "1;smithi151=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:45.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[29291]: Saving service iscsi.datapool spec with placement smithi151=iscsi.a;count:1 2024-03-12T03:13:45.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:45.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[40484]: osdmap e66: 8 total, 8 up, 8 in 2024-03-12T03:13:45.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:13:45.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:13:45.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[40484]: pgmap v197: 132 pgs: 14 creating+peering, 72 active+clean, 46 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-12T03:13:45.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[40484]: from='client.24418 v1:172.21.15.151:0/3868626772' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.151", "placement": "1;smithi151=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:45.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[40484]: Saving service iscsi.datapool spec with placement smithi151=iscsi.a;count:1 2024-03-12T03:13:45.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:44 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:45.448 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:13:46.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:13:46.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:13:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:45 smithi094 ceph-mon[29291]: osdmap e67: 8 total, 8 up, 8 in 2024-03-12T03:13:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:13:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:13:46.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:13:46.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:13:46.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:45 smithi094 ceph-mon[40484]: osdmap e67: 8 total, 8 up, 8 in 2024-03-12T03:13:46.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:13:46.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:13:46.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:13:46.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:13:46.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:45 smithi151 ceph-mon[32923]: osdmap e67: 8 total, 8 up, 8 in 2024-03-12T03:13:46.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:13:46.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:13:47.103 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:13:47.104 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:13:47.104 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:46 smithi151 ceph-mon[32923]: osdmap e68: 8 total, 8 up, 8 in 2024-03-12T03:13:47.104 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:46 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:47.104 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:46 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:47.104 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:46 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:47.104 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:46 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:13:47.104 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:46 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-12T03:13:47.104 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:46 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:47.104 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:46 smithi151 ceph-mon[32923]: Deploying daemon iscsi.iscsi.a on smithi151 2024-03-12T03:13:47.104 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:46 smithi151 ceph-mon[32923]: pgmap v200: 132 pgs: 9 creating+peering, 107 active+clean, 16 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-12T03:13:47.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.100+0000 7f159d71c700 1 -- 172.21.15.151:0/2575964844 >> v1:172.21.15.151:6789/0 conn(0x7f1598108c30 legacy=0x7f159810b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:47.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.100+0000 7f159d71c700 1 -- 172.21.15.151:0/2575964844 shutdown_connections 2024-03-12T03:13:47.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.100+0000 7f159d71c700 1 -- 172.21.15.151:0/2575964844 >> 172.21.15.151:0/2575964844 conn(0x7f15980fb850 msgr2=0x7f15980fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:47.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.100+0000 7f159d71c700 1 -- 172.21.15.151:0/2575964844 shutdown_connections 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.101+0000 7f159d71c700 1 -- 172.21.15.151:0/2575964844 wait complete. 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.101+0000 7f159d71c700 1 Processor -- start 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.101+0000 7f159d71c700 1 -- start start 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f159d71c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1598197f40 con 0x7f1598102080 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f159d71c700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15981a3730 con 0x7f1598105000 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f159d71c700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15981a4910 con 0x7f1598108c30 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f15967fc700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f1598105000 0x7f159819d8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:51344/0 (socket says 172.21.15.151:51344) 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f15967fc700 1 -- 172.21.15.151:0/1817704559 learned_addr learned my addr 172.21.15.151:0/1817704559 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903948095 0 0) 0x7f15981a3730 con 0x7f1598105000 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15800035b0 con 0x7f1598105000 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063289151 0 0) 0x7f1598197f40 con 0x7f1598102080 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15981a3730 con 0x7f1598102080 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426962536 0 0) 0x7f15981a4910 con 0x7f1598108c30 2024-03-12T03:13:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1598197f40 con 0x7f1598108c30 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3846750680 0 0) 0x7f15981a3730 con 0x7f1598102080 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15981a4910 con 0x7f1598102080 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f15880018b0 con 0x7f1598102080 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 281260559 0 0) 0x7f15800035b0 con 0x7f1598105000 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15981a3730 con 0x7f1598105000 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3129610907 0 0) 0x7f1598197f40 con 0x7f1598108c30 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15800035b0 con 0x7f1598108c30 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f158c0018b0 con 0x7f1598105000 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.102+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f15900017b0 con 0x7f1598108c30 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.103+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3999281912 0 0) 0x7f15981a4910 con 0x7f1598102080 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.103+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 >> v1:172.21.15.94:6790/0 conn(0x7f1598108c30 legacy=0x7f15981a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.103+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 >> v1:172.21.15.151:6789/0 conn(0x7f1598105000 legacy=0x7f159819d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.103+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15981a5b30 con 0x7f1598102080 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.103+0000 7f159d71c700 1 -- 172.21.15.151:0/1817704559 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15981a2780 con 0x7f1598102080 2024-03-12T03:13:47.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.103+0000 7f159d71c700 1 -- 172.21.15.151:0/1817704559 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15981a2cc0 con 0x7f1598102080 2024-03-12T03:13:47.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.103+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f1588003b30 con 0x7f1598102080 2024-03-12T03:13:47.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.103+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1588004970 con 0x7f1598102080 2024-03-12T03:13:47.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.104+0000 7f159d71c700 1 -- 172.21.15.151:0/1817704559 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1584005120 con 0x7f1598102080 2024-03-12T03:13:47.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.104+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 15) v1 ==== 91719+0+0 (unknown 3315827787 0 0) 0x7f158801b240 con 0x7f1598102080 2024-03-12T03:13:47.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.105+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (unknown 2467756649 0 0) 0x7f1588087c70 con 0x7f1598102080 2024-03-12T03:13:47.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.109+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f15880555f0 con 0x7f1598102080 2024-03-12T03:13:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:13:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:13:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[29291]: osdmap e68: 8 total, 8 up, 8 in 2024-03-12T03:13:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:13:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-12T03:13:47.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:47.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[29291]: Deploying daemon iscsi.iscsi.a on smithi151 2024-03-12T03:13:47.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[29291]: pgmap v200: 132 pgs: 9 creating+peering, 107 active+clean, 16 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-12T03:13:47.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/806874325' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:13:47.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/307000586' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:13:47.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[40484]: osdmap e68: 8 total, 8 up, 8 in 2024-03-12T03:13:47.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:47.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:47.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:47.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:13:47.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-12T03:13:47.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:47.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[40484]: Deploying daemon iscsi.iscsi.a on smithi151 2024-03-12T03:13:47.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:46 smithi094 ceph-mon[40484]: pgmap v200: 132 pgs: 9 creating+peering, 107 active+clean, 16 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-12T03:13:47.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.327+0000 7f159d71c700 1 -- 172.21.15.151:0/1817704559 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f1584000b90 con 0x7f158006ef00 2024-03-12T03:13:47.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.333+0000 7f157ffff700 1 -- 172.21.15.151:0/1817704559 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (unknown 0 0 1342662408) 0x7f1584000b90 con 0x7f158006ef00 2024-03-12T03:13:47.336 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled prometheus update... 2024-03-12T03:13:47.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.335+0000 7f159d71c700 1 -- 172.21.15.151:0/1817704559 >> v1:172.21.15.94:6800/1299282424 conn(0x7f158006ef00 legacy=0x7f15800713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:47.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.335+0000 7f159d71c700 1 -- 172.21.15.151:0/1817704559 >> v1:172.21.15.94:6789/0 conn(0x7f1598102080 legacy=0x7f15981973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:47.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.335+0000 7f159d71c700 1 -- 172.21.15.151:0/1817704559 shutdown_connections 2024-03-12T03:13:47.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.335+0000 7f159d71c700 1 -- 172.21.15.151:0/1817704559 >> 172.21.15.151:0/1817704559 conn(0x7f15980fb850 msgr2=0x7f15981004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:47.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.336+0000 7f159d71c700 1 -- 172.21.15.151:0/1817704559 shutdown_connections 2024-03-12T03:13:47.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:47.336+0000 7f159d71c700 1 -- 172.21.15.151:0/1817704559 wait complete. 2024-03-12T03:13:48.024 DEBUG:teuthology.orchestra.run.smithi151:prometheus.a> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-12T03:13:48.026 INFO:tasks.cephadm:Adding node-exporter.a on smithi094 2024-03-12T03:13:48.027 INFO:tasks.cephadm:Adding node-exporter.b on smithi151 2024-03-12T03:13:48.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '2;smithi094=a;smithi151=b' 2024-03-12T03:13:48.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:47 smithi094 ceph-mon[29291]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:13:48.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:47 smithi094 ceph-mon[29291]: Cluster is now healthy 2024-03-12T03:13:48.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:47 smithi094 ceph-mon[29291]: from='client.14541 v1:172.21.15.151:0/1817704559' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:48.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:47 smithi094 ceph-mon[29291]: Saving service prometheus spec with placement smithi151=a;count:1 2024-03-12T03:13:48.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:47 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:48.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:47 smithi094 ceph-mon[29291]: osdmap e69: 8 total, 8 up, 8 in 2024-03-12T03:13:48.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:47 smithi094 ceph-mon[40484]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:13:48.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:47 smithi094 ceph-mon[40484]: Cluster is now healthy 2024-03-12T03:13:48.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:47 smithi094 ceph-mon[40484]: from='client.14541 v1:172.21.15.151:0/1817704559' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:48.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:47 smithi094 ceph-mon[40484]: Saving service prometheus spec with placement smithi151=a;count:1 2024-03-12T03:13:48.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:47 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:48.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:47 smithi094 ceph-mon[40484]: osdmap e69: 8 total, 8 up, 8 in 2024-03-12T03:13:48.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:47 smithi151 ceph-mon[32923]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:13:48.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:47 smithi151 ceph-mon[32923]: Cluster is now healthy 2024-03-12T03:13:48.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:47 smithi151 ceph-mon[32923]: from='client.14541 v1:172.21.15.151:0/1817704559' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:48.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:47 smithi151 ceph-mon[32923]: Saving service prometheus spec with placement smithi151=a;count:1 2024-03-12T03:13:48.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:47 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:48.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:47 smithi151 ceph-mon[32923]: osdmap e69: 8 total, 8 up, 8 in 2024-03-12T03:13:48.843 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:13:49.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:48 smithi094 ceph-mon[29291]: pgmap v202: 132 pgs: 9 creating+peering, 118 active+clean, 5 unknown; 579 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 2.5 KiB/s wr, 93 op/s 2024-03-12T03:13:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:48 smithi094 ceph-mon[40484]: pgmap v202: 132 pgs: 9 creating+peering, 118 active+clean, 5 unknown; 579 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 2.5 KiB/s wr, 93 op/s 2024-03-12T03:13:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:48 smithi151 ceph-mon[32923]: pgmap v202: 132 pgs: 9 creating+peering, 118 active+clean, 5 unknown; 579 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 2.5 KiB/s wr, 93 op/s 2024-03-12T03:13:49.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:13:49 smithi151 systemd[1]: Starting Ceph iscsi.iscsi.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:13:50.034 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:13:49 smithi151 podman[60078]: 2024-03-12 03:13:49.626889005 +0000 UTC m=+0.017743141 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:13:50.319 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:13:50 smithi151 podman[60078]: 2024-03-12 03:13:50.191699601 +0000 UTC m=+0.582553795 container create d08b29abd1003b96824f38a0c4dfe878a098451967303ed8c37496be3fd361bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-12T03:13:50.641 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:13:50 smithi151 podman[60078]: 2024-03-12 03:13:50.411070246 +0000 UTC m=+0.801924385 container init d08b29abd1003b96824f38a0c4dfe878a098451967303ed8c37496be3fd361bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-12T03:13:50.641 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:13:50 smithi151 podman[60078]: 2024-03-12 03:13:50.414273808 +0000 UTC m=+0.805127936 container start d08b29abd1003b96824f38a0c4dfe878a098451967303ed8c37496be3fd361bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e) 2024-03-12T03:13:50.641 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:13:50 smithi151 bash[60078]: d08b29abd1003b96824f38a0c4dfe878a098451967303ed8c37496be3fd361bc 2024-03-12T03:13:50.641 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:13:50 smithi151 systemd[1]: Started Ceph iscsi.iscsi.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:13:50.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:13:50 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug Started the configuration object watcher 2024-03-12T03:13:50.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:13:50 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug Checking for config object changes every 1s 2024-03-12T03:13:50.933 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:13:50 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug Processing osd blocklist entries for this node 2024-03-12T03:13:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.066+0000 7f50c64e1700 1 -- 172.21.15.151:0/2004292314 >> v1:172.21.15.151:6789/0 conn(0x7f50c0105000 legacy=0x7f50c01073f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.066+0000 7f50c64e1700 1 -- 172.21.15.151:0/2004292314 shutdown_connections 2024-03-12T03:13:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.066+0000 7f50c64e1700 1 -- 172.21.15.151:0/2004292314 >> 172.21.15.151:0/2004292314 conn(0x7f50c00fb870 msgr2=0x7f50c00fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.066+0000 7f50c64e1700 1 -- 172.21.15.151:0/2004292314 shutdown_connections 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.066+0000 7f50c64e1700 1 -- 172.21.15.151:0/2004292314 wait complete. 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.067+0000 7f50c64e1700 1 Processor -- start 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.067+0000 7f50c64e1700 1 -- start start 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.067+0000 7f50c64e1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f50c0197f30 con 0x7f50c0102110 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.067+0000 7f50c64e1700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f50c01a3720 con 0x7f50c0105000 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.067+0000 7f50c64e1700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f50c01a4900 con 0x7f50c0108c60 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.067+0000 7f50bf7fe700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f50c0105000 0x7f50c019d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:51452/0 (socket says 172.21.15.151:51452) 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.067+0000 7f50bf7fe700 1 -- 172.21.15.151:0/915220473 learned_addr learned my addr 172.21.15.151:0/915220473 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71917326 0 0) 0x7f50c01a3720 con 0x7f50c0105000 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50a80035b0 con 0x7f50c0105000 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212813697 0 0) 0x7f50c01a4900 con 0x7f50c0108c60 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c01a3720 con 0x7f50c0108c60 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78942650 0 0) 0x7f50c0197f30 con 0x7f50c0102110 2024-03-12T03:13:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c01a4900 con 0x7f50c0102110 2024-03-12T03:13:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 651518547 0 0) 0x7f50a80035b0 con 0x7f50c0105000 2024-03-12T03:13:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50c0197f30 con 0x7f50c0105000 2024-03-12T03:13:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f50b0001970 con 0x7f50c0105000 2024-03-12T03:13:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4156484412 0 0) 0x7f50c01a3720 con 0x7f50c0108c60 2024-03-12T03:13:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50a80035b0 con 0x7f50c0108c60 2024-03-12T03:13:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f50b8001bb0 con 0x7f50c0108c60 2024-03-12T03:13:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2850781838 0 0) 0x7f50c0197f30 con 0x7f50c0105000 2024-03-12T03:13:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 >> v1:172.21.15.94:6790/0 conn(0x7f50c0108c60 legacy=0x7f50c01a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 >> v1:172.21.15.94:6789/0 conn(0x7f50c0102110 legacy=0x7f50c01973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c01a5b20 con 0x7f50c0105000 2024-03-12T03:13:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50c64e1700 1 -- 172.21.15.151:0/915220473 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c01a4b30 con 0x7f50c0105000 2024-03-12T03:13:51.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50c64e1700 1 -- 172.21.15.151:0/915220473 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c01a5070 con 0x7f50c0105000 2024-03-12T03:13:51.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f50b0001c80 con 0x7f50c0105000 2024-03-12T03:13:51.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.068+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f50b0004da0 con 0x7f50c0105000 2024-03-12T03:13:51.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.069+0000 7f50c64e1700 1 -- 172.21.15.151:0/915220473 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ac005150 con 0x7f50c0105000 2024-03-12T03:13:51.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.070+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 15) v1 ==== 91719+0+0 (unknown 3315827787 0 0) 0x7f50b0004d80 con 0x7f50c0105000 2024-03-12T03:13:51.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.070+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (unknown 2765393099 0 0) 0x7f50b0089070 con 0x7f50c0105000 2024-03-12T03:13:51.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.075+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f50b0057840 con 0x7f50c0105000 2024-03-12T03:13:51.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:13:51 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug Reading the configuration object to update local LIO configuration 2024-03-12T03:13:51.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:13:51 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug Configuration does not have an entry for this host(smithi151.front.sepia.ceph.com) - nothing to define to LIO 2024-03-12T03:13:51.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:13:51 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-12T03:13:51.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.313+0000 7f50c64e1700 1 -- 172.21.15.151:0/915220473 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi094=a;smithi151=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f50ac000bc0 con 0x7f50a806eb80 2024-03-12T03:13:51.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.320+0000 7f50bd7fa700 1 -- 172.21.15.151:0/915220473 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (unknown 0 0 240551134) 0x7f50ac000bc0 con 0x7f50a806eb80 2024-03-12T03:13:51.323 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled node-exporter update... 2024-03-12T03:13:51.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.322+0000 7f50c64e1700 1 -- 172.21.15.151:0/915220473 >> v1:172.21.15.94:6800/1299282424 conn(0x7f50a806eb80 legacy=0x7f50a8071040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:51.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.322+0000 7f50c64e1700 1 -- 172.21.15.151:0/915220473 >> v1:172.21.15.151:6789/0 conn(0x7f50c0105000 legacy=0x7f50c019d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:51.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.322+0000 7f50c64e1700 1 -- 172.21.15.151:0/915220473 shutdown_connections 2024-03-12T03:13:51.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.322+0000 7f50c64e1700 1 -- 172.21.15.151:0/915220473 >> 172.21.15.151:0/915220473 conn(0x7f50c00fb870 msgr2=0x7f50c01053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:51.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.322+0000 7f50c64e1700 1 -- 172.21.15.151:0/915220473 shutdown_connections 2024-03-12T03:13:51.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:51.322+0000 7f50c64e1700 1 -- 172.21.15.151:0/915220473 wait complete. 2024-03-12T03:13:51.514 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:51 smithi151 ceph-mon[32923]: pgmap v203: 132 pgs: 132 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 4.3 KiB/s wr, 220 op/s 2024-03-12T03:13:51.515 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:51 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.515 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:51 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.515 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:51 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.515 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:51 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.515 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:51 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:51.515 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/319366409' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:13:51.515 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:51 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[29291]: pgmap v203: 132 pgs: 132 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 4.3 KiB/s wr, 220 op/s 2024-03-12T03:13:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/319366409' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:13:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[40484]: pgmap v203: 132 pgs: 132 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 4.3 KiB/s wr, 220 op/s 2024-03-12T03:13:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:13:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/319366409' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:13:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:51 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:52.084 DEBUG:teuthology.orchestra.run.smithi094:node-exporter.a> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-12T03:13:52.087 DEBUG:teuthology.orchestra.run.smithi151:node-exporter.b> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-12T03:13:52.089 INFO:tasks.cephadm:Adding alertmanager.a on smithi094 2024-03-12T03:13:52.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi094=a' 2024-03-12T03:13:52.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:52 smithi094 ceph-mon[29291]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-12T03:13:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:52 smithi094 ceph-mon[29291]: from='client.24475 v1:172.21.15.151:0/915220473' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi094=a;smithi151=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:52 smithi094 ceph-mon[29291]: Saving service node-exporter spec with placement smithi094=a;smithi151=b;count:2 2024-03-12T03:13:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:52 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:52 smithi094 ceph-mon[40484]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-12T03:13:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:52 smithi094 ceph-mon[40484]: from='client.24475 v1:172.21.15.151:0/915220473' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi094=a;smithi151=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:52 smithi094 ceph-mon[40484]: Saving service node-exporter spec with placement smithi094=a;smithi151=b;count:2 2024-03-12T03:13:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:52 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:52 smithi151 ceph-mon[32923]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-12T03:13:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:52 smithi151 ceph-mon[32923]: from='client.24475 v1:172.21.15.151:0/915220473' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi094=a;smithi151=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:52 smithi151 ceph-mon[32923]: Saving service node-exporter spec with placement smithi094=a;smithi151=b;count:2 2024-03-12T03:13:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:52 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:53.012 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:13:53.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:53 smithi094 ceph-mon[40484]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 4.7 KiB/s wr, 214 op/s 2024-03-12T03:13:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:53 smithi094 ceph-mon[40484]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-12T03:13:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:53 smithi094 ceph-mon[29291]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 4.7 KiB/s wr, 214 op/s 2024-03-12T03:13:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:53 smithi094 ceph-mon[29291]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-12T03:13:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:53 smithi151 ceph-mon[32923]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 4.7 KiB/s wr, 214 op/s 2024-03-12T03:13:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:53 smithi151 ceph-mon[32923]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.647+0000 7efc82116700 1 -- 172.21.15.151:0/4147968063 >> v1:172.21.15.151:6789/0 conn(0x7efc7c106010 legacy=0x7efc7c108460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.647+0000 7efc82116700 1 -- 172.21.15.151:0/4147968063 shutdown_connections 2024-03-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.647+0000 7efc82116700 1 -- 172.21.15.151:0/4147968063 >> 172.21.15.151:0/4147968063 conn(0x7efc7c0fd180 msgr2=0x7efc7c0ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.647+0000 7efc82116700 1 -- 172.21.15.151:0/4147968063 shutdown_connections 2024-03-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.647+0000 7efc82116700 1 -- 172.21.15.151:0/4147968063 wait complete. 2024-03-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.647+0000 7efc82116700 1 Processor -- start 2024-03-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.647+0000 7efc82116700 1 -- start start 2024-03-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc82116700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efc7c197d00 con 0x7efc7c101be0 2024-03-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc82116700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efc7c1a34f0 con 0x7efc7c109d50 2024-03-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc82116700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efc7c1a46d0 con 0x7efc7c106010 2024-03-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc7bfff700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7efc7c109d50 0x7efc7c1a0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:33332/0 (socket says 172.21.15.151:33332) 2024-03-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc7bfff700 1 -- 172.21.15.151:0/4054687855 learned_addr learned my addr 172.21.15.151:0/4054687855 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772279212 0 0) 0x7efc7c1a34f0 con 0x7efc7c109d50 2024-03-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc640035b0 con 0x7efc7c109d50 2024-03-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708147495 0 0) 0x7efc7c197d00 con 0x7efc7c101be0 2024-03-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled alertmanager update... 2024-03-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc7c1a34f0 con 0x7efc7c101be0 2024-03-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168692434 0 0) 0x7efc7c1a46d0 con 0x7efc7c106010 2024-03-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc7c197d00 con 0x7efc7c106010 2024-03-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2740557747 0 0) 0x7efc640035b0 con 0x7efc7c109d50 2024-03-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc7c1a46d0 con 0x7efc7c109d50 2024-03-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7efc74001c10 con 0x7efc7c109d50 2024-03-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3342759307 0 0) 0x7efc7c197d00 con 0x7efc7c106010 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc640035b0 con 0x7efc7c106010 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7efc70001430 con 0x7efc7c106010 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 111110377 0 0) 0x7efc7c1a46d0 con 0x7efc7c109d50 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 >> v1:172.21.15.94:6790/0 conn(0x7efc7c106010 legacy=0x7efc7c19d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 >> v1:172.21.15.94:6789/0 conn(0x7efc7c101be0 legacy=0x7efc7c1971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c1a58f0 con 0x7efc7c109d50 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc82116700 1 -- 172.21.15.151:0/4054687855 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c1a2540 con 0x7efc7c109d50 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc82116700 1 -- 172.21.15.151:0/4054687855 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c1a2a80 con 0x7efc7c109d50 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7efc74003eb0 con 0x7efc7c109d50 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.648+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7efc74005f30 con 0x7efc7c109d50 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.649+0000 7efc82116700 1 -- 172.21.15.151:0/4054687855 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc68005150 con 0x7efc7c109d50 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.650+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 16) v1 ==== 91770+0+0 (unknown 3875668065 0 0) 0x7efc74003a20 con 0x7efc7c109d50 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.650+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (unknown 2765393099 0 0) 0x7efc74089370 con 0x7efc7c109d50 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.655+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7efc74056bd0 con 0x7efc7c109d50 2024-03-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.875+0000 7efc82116700 1 -- 172.21.15.151:0/4054687855 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi094=a", "target": ["mon-mgr", ""]}) v1 -- 0x7efc68000bc0 con 0x7efc6406e9c0 2024-03-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.883+0000 7efc78ff9700 1 -- 172.21.15.151:0/4054687855 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (unknown 0 0 1850065467) 0x7efc68000bc0 con 0x7efc6406e9c0 2024-03-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.885+0000 7efc82116700 1 -- 172.21.15.151:0/4054687855 >> v1:172.21.15.94:6800/1299282424 conn(0x7efc6406e9c0 legacy=0x7efc64070e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.885+0000 7efc82116700 1 -- 172.21.15.151:0/4054687855 >> v1:172.21.15.151:6789/0 conn(0x7efc7c109d50 legacy=0x7efc7c1a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.885+0000 7efc82116700 1 -- 172.21.15.151:0/4054687855 shutdown_connections 2024-03-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.885+0000 7efc82116700 1 -- 172.21.15.151:0/4054687855 >> 172.21.15.151:0/4054687855 conn(0x7efc7c0fd180 msgr2=0x7efc7c108460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.885+0000 7efc82116700 1 -- 172.21.15.151:0/4054687855 shutdown_connections 2024-03-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:54.885+0000 7efc82116700 1 -- 172.21.15.151:0/4054687855 wait complete. 2024-03-12T03:13:55.528 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:55 smithi151 ceph-mon[32923]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 4.0 KiB/s wr, 188 op/s 2024-03-12T03:13:55.528 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:55 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:55.528 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:55 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:55.528 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:55 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:55.528 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:55 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:55.528 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:55 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:55.528 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:55 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[29291]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 4.0 KiB/s wr, 188 op/s 2024-03-12T03:13:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:55.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:55.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[40484]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 4.0 KiB/s wr, 188 op/s 2024-03-12T03:13:55.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:55.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:55.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:55.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:13:55.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:13:55.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:55 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:55.799 DEBUG:teuthology.orchestra.run.smithi094:alertmanager.a> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-12T03:13:55.801 INFO:tasks.cephadm:Adding grafana.a on smithi151 2024-03-12T03:13:55.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi151=a' 2024-03-12T03:13:56.053 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:13:56.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:56 smithi094 ceph-mon[40484]: from='client.24481 v1:172.21.15.151:0/4054687855' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi094=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:56 smithi094 ceph-mon[40484]: Saving service alertmanager spec with placement smithi094=a;count:1 2024-03-12T03:13:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:56 smithi094 ceph-mon[40484]: Deploying daemon prometheus.a on smithi151 2024-03-12T03:13:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:56 smithi094 ceph-mon[29291]: from='client.24481 v1:172.21.15.151:0/4054687855' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi094=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:56 smithi094 ceph-mon[29291]: Saving service alertmanager spec with placement smithi094=a;count:1 2024-03-12T03:13:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:56 smithi094 ceph-mon[29291]: Deploying daemon prometheus.a on smithi151 2024-03-12T03:13:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:56 smithi151 ceph-mon[32923]: from='client.24481 v1:172.21.15.151:0/4054687855' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi094=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:56 smithi151 ceph-mon[32923]: Saving service alertmanager spec with placement smithi094=a;count:1 2024-03-12T03:13:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:56 smithi151 ceph-mon[32923]: Deploying daemon prometheus.a on smithi151 2024-03-12T03:13:57.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:57 smithi094 ceph-mon[40484]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 3.4 KiB/s wr, 158 op/s 2024-03-12T03:13:57.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:57 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:57 smithi094 ceph-mon[29291]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 3.4 KiB/s wr, 158 op/s 2024-03-12T03:13:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:57 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:57 smithi151 ceph-mon[32923]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 3.4 KiB/s wr, 158 op/s 2024-03-12T03:13:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:57 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:57.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.876+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/3533574746 >> v1:172.21.15.151:6789/0 conn(0x7f4fc4108c40 legacy=0x7f4fc410b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:57.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.876+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/3533574746 shutdown_connections 2024-03-12T03:13:57.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.876+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/3533574746 >> 172.21.15.151:0/3533574746 conn(0x7f4fc40fc070 msgr2=0x7f4fc40fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:57.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.877+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/3533574746 shutdown_connections 2024-03-12T03:13:57.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.877+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/3533574746 wait complete. 2024-03-12T03:13:57.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.877+0000 7f4fcbcc4700 1 Processor -- start 2024-03-12T03:13:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.877+0000 7f4fcbcc4700 1 -- start start 2024-03-12T03:13:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.877+0000 7f4fcbcc4700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4fc4197eb0 con 0x7f4fc4104f00 2024-03-12T03:13:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.877+0000 7f4fcbcc4700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4fc41a36a0 con 0x7f4fc4108c40 2024-03-12T03:13:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.877+0000 7f4fcbcc4700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4fc41a4880 con 0x7f4fc4100ad0 2024-03-12T03:13:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.877+0000 7f4fca261700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4fc4108c40 0x7f4fc41a0f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:33356/0 (socket says 172.21.15.151:33356) 2024-03-12T03:13:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.877+0000 7f4fca261700 1 -- 172.21.15.151:0/2399776182 learned_addr learned my addr 172.21.15.151:0/2399776182 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:13:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236965797 0 0) 0x7f4fc41a36a0 con 0x7f4fc4108c40 2024-03-12T03:13:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fb00035b0 con 0x7f4fc4108c40 2024-03-12T03:13:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112475483 0 0) 0x7f4fc41a4880 con 0x7f4fc4100ad0 2024-03-12T03:13:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc41a36a0 con 0x7f4fc4100ad0 2024-03-12T03:13:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208056901 0 0) 0x7f4fc4197eb0 con 0x7f4fc4104f00 2024-03-12T03:13:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc41a4880 con 0x7f4fc4104f00 2024-03-12T03:13:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 456021128 0 0) 0x7f4fb00035b0 con 0x7f4fc4108c40 2024-03-12T03:13:57.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fc4197eb0 con 0x7f4fc4108c40 2024-03-12T03:13:57.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f4fbc0017b0 con 0x7f4fc4108c40 2024-03-12T03:13:57.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3215705760 0 0) 0x7f4fc41a36a0 con 0x7f4fc4100ad0 2024-03-12T03:13:57.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fb00035b0 con 0x7f4fc4100ad0 2024-03-12T03:13:57.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f4fc00018b0 con 0x7f4fc4100ad0 2024-03-12T03:13:57.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2286898589 0 0) 0x7f4fc41a4880 con 0x7f4fc4104f00 2024-03-12T03:13:57.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fc41a36a0 con 0x7f4fc4104f00 2024-03-12T03:13:57.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f4fb40017f0 con 0x7f4fc4104f00 2024-03-12T03:13:57.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 444579055 0 0) 0x7f4fc4197eb0 con 0x7f4fc4108c40 2024-03-12T03:13:57.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 >> v1:172.21.15.94:6790/0 conn(0x7f4fc4100ad0 legacy=0x7f4fc4197360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:57.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 >> v1:172.21.15.94:6789/0 conn(0x7f4fc4104f00 legacy=0x7f4fc419d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:57.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc41a5aa0 con 0x7f4fc4108c40 2024-03-12T03:13:57.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/2399776182 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc41a4ab0 con 0x7f4fc4108c40 2024-03-12T03:13:57.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/2399776182 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc41a4ff0 con 0x7f4fc4108c40 2024-03-12T03:13:57.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f4fbc001a10 con 0x7f4fc4108c40 2024-03-12T03:13:57.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.878+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f4fbc005250 con 0x7f4fc4108c40 2024-03-12T03:13:57.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.879+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/2399776182 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa8005150 con 0x7f4fc4108c40 2024-03-12T03:13:57.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.880+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 16) v1 ==== 91770+0+0 (unknown 3875668065 0 0) 0x7f4fbc003e30 con 0x7f4fc4108c40 2024-03-12T03:13:57.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.880+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (unknown 2765393099 0 0) 0x7f4fbc0894b0 con 0x7f4fc4108c40 2024-03-12T03:13:57.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:57.885+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f4fbc056d90 con 0x7f4fc4108c40 2024-03-12T03:13:58.132 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:58.129+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/2399776182 --> v1:172.21.15.94:6800/1299282424 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fa8000bc0 con 0x7f4fb006eea0 2024-03-12T03:13:58.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:58.137+0000 7f4fbaffd700 1 -- 172.21.15.151:0/2399776182 <== mgr.14152 v1:172.21.15.94:6800/1299282424 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (unknown 0 0 664801700) 0x7f4fa8000bc0 con 0x7f4fb006eea0 2024-03-12T03:13:58.140 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled grafana update... 2024-03-12T03:13:58.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:58.140+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/2399776182 >> v1:172.21.15.94:6800/1299282424 conn(0x7f4fb006eea0 legacy=0x7f4fb0071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:58.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:58.140+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/2399776182 >> v1:172.21.15.151:6789/0 conn(0x7f4fc4108c40 legacy=0x7f4fc41a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:13:58.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:58.140+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/2399776182 shutdown_connections 2024-03-12T03:13:58.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:58.140+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/2399776182 >> 172.21.15.151:0/2399776182 conn(0x7f4fc40fc070 msgr2=0x7f4fc4069760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:13:58.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:58.140+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/2399776182 shutdown_connections 2024-03-12T03:13:58.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:13:58.140+0000 7f4fcbcc4700 1 -- 172.21.15.151:0/2399776182 wait complete. 2024-03-12T03:13:59.337 DEBUG:teuthology.orchestra.run.smithi151:grafana.a> sudo journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@grafana.a.service 2024-03-12T03:13:59.339 INFO:tasks.cephadm:Setting up client nodes... 2024-03-12T03:13:59.339 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T03:13:59.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:59 smithi094 ceph-mon[29291]: from='client.24487 v1:172.21.15.151:0/2399776182' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:59 smithi094 ceph-mon[29291]: Saving service grafana spec with placement smithi151=a;count:1 2024-03-12T03:13:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:59 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:13:59 smithi094 ceph-mon[29291]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 2.7 KiB/s wr, 115 op/s 2024-03-12T03:13:59.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:59 smithi094 ceph-mon[40484]: from='client.24487 v1:172.21.15.151:0/2399776182' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:59.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:59 smithi094 ceph-mon[40484]: Saving service grafana spec with placement smithi151=a;count:1 2024-03-12T03:13:59.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:59 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:59.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:13:59 smithi094 ceph-mon[40484]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 2.7 KiB/s wr, 115 op/s 2024-03-12T03:13:59.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:59 smithi151 ceph-mon[32923]: from='client.24487 v1:172.21.15.151:0/2399776182' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:13:59.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:59 smithi151 ceph-mon[32923]: Saving service grafana spec with placement smithi151=a;count:1 2024-03-12T03:13:59.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:59 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:13:59.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:13:59 smithi151 ceph-mon[32923]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 2.7 KiB/s wr, 115 op/s 2024-03-12T03:13:59.588 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:01.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:14:00 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:14:01.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.394+0000 7ff0de5d8700 1 -- 172.21.15.94:0/1125808817 >> v1:172.21.15.94:6790/0 conn(0x7ff0d8106010 legacy=0x7ff0d8108460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:01.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.394+0000 7ff0de5d8700 1 -- 172.21.15.94:0/1125808817 shutdown_connections 2024-03-12T03:14:01.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.394+0000 7ff0de5d8700 1 -- 172.21.15.94:0/1125808817 >> 172.21.15.94:0/1125808817 conn(0x7ff0d80fd180 msgr2=0x7ff0d80ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:01.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.394+0000 7ff0de5d8700 1 -- 172.21.15.94:0/1125808817 shutdown_connections 2024-03-12T03:14:01.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.395+0000 7ff0de5d8700 1 -- 172.21.15.94:0/1125808817 wait complete. 2024-03-12T03:14:01.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.395+0000 7ff0de5d8700 1 Processor -- start 2024-03-12T03:14:01.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.395+0000 7ff0de5d8700 1 -- start start 2024-03-12T03:14:01.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.395+0000 7ff0de5d8700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0d81a4260 con 0x7ff0d8109d50 2024-03-12T03:14:01.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.395+0000 7ff0de5d8700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0d81a54a0 con 0x7ff0d8101be0 2024-03-12T03:14:01.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.395+0000 7ff0de5d8700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0d81a66e0 con 0x7ff0d8106010 2024-03-12T03:14:01.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d77fe700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7ff0d8106010 0x7ff0d819f040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:38022/0 (socket says 172.21.15.94:38022) 2024-03-12T03:14:01.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0dcb75700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff0d8109d50 0x7ff0d81a2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51768/0 (socket says 172.21.15.94:51768) 2024-03-12T03:14:01.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d77fe700 1 -- 172.21.15.94:0/2932737389 learned_addr learned my addr 172.21.15.94:0/2932737389 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:01.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891455383 0 0) 0x7ff0d81a66e0 con 0x7ff0d8106010 2024-03-12T03:14:01.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c00035b0 con 0x7ff0d8106010 2024-03-12T03:14:01.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217479962 0 0) 0x7ff0d81a4260 con 0x7ff0d8109d50 2024-03-12T03:14:01.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d81a66e0 con 0x7ff0d8109d50 2024-03-12T03:14:01.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500529109 0 0) 0x7ff0d81a54a0 con 0x7ff0d8101be0 2024-03-12T03:14:01.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d81a4260 con 0x7ff0d8101be0 2024-03-12T03:14:01.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1076405431 0 0) 0x7ff0c00035b0 con 0x7ff0d8106010 2024-03-12T03:14:01.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0d81a54a0 con 0x7ff0d8106010 2024-03-12T03:14:01.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7ff0cc004510 con 0x7ff0d8106010 2024-03-12T03:14:01.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1964853418 0 0) 0x7ff0d81a66e0 con 0x7ff0d8109d50 2024-03-12T03:14:01.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0c00035b0 con 0x7ff0d8109d50 2024-03-12T03:14:01.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7ff0d0001c30 con 0x7ff0d8109d50 2024-03-12T03:14:01.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2410869591 0 0) 0x7ff0d81a4260 con 0x7ff0d8101be0 2024-03-12T03:14:01.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0d81a66e0 con 0x7ff0d8101be0 2024-03-12T03:14:01.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7ff0c8001870 con 0x7ff0d8101be0 2024-03-12T03:14:01.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1860115664 0 0) 0x7ff0d81a54a0 con 0x7ff0d8106010 2024-03-12T03:14:01.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 >> v1:172.21.15.151:6789/0 conn(0x7ff0d8101be0 legacy=0x7ff0d8075aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:01.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 >> v1:172.21.15.94:6789/0 conn(0x7ff0d8109d50 legacy=0x7ff0d81a2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:01.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d81a7920 con 0x7ff0d8106010 2024-03-12T03:14:01.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0de5d8700 1 -- 172.21.15.94:0/2932737389 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d81a4490 con 0x7ff0d8106010 2024-03-12T03:14:01.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.396+0000 7ff0de5d8700 1 -- 172.21.15.94:0/2932737389 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d81a49d0 con 0x7ff0d8106010 2024-03-12T03:14:01.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.397+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7ff0cc001ae0 con 0x7ff0d8106010 2024-03-12T03:14:01.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.397+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7ff0cc005100 con 0x7ff0d8106010 2024-03-12T03:14:01.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.398+0000 7ff0de5d8700 1 -- 172.21.15.94:0/2932737389 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d810e3b0 con 0x7ff0d8106010 2024-03-12T03:14:01.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.398+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 16) v1 ==== 91770+0+0 (unknown 3875668065 0 0) 0x7ff0cc003710 con 0x7ff0d8106010 2024-03-12T03:14:01.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.398+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (unknown 2765393099 0 0) 0x7ff0cc089250 con 0x7ff0d8106010 2024-03-12T03:14:01.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.403+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7ff0cc056b50 con 0x7ff0d8106010 2024-03-12T03:14:01.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.679+0000 7ff0de5d8700 1 -- 172.21.15.94:0/2932737389 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7ff0d810dce0 con 0x7ff0d8106010 2024-03-12T03:14:01.682 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:01 smithi094 ceph-mon[40484]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 2.3 KiB/s wr, 100 op/s 2024-03-12T03:14:01.682 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:01 smithi094 ceph-mon[29291]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 2.3 KiB/s wr, 100 op/s 2024-03-12T03:14:01.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.683+0000 7ff0d57fa700 1 -- 172.21.15.94:0/2932737389 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (unknown 57899324 0 1431720134) 0x7ff0cc056450 con 0x7ff0d8106010 2024-03-12T03:14:01.686 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-12T03:14:01.686 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB5yO9lG+iYKBAAac2Ew66ZB2us4u02JQxf4Q== 2024-03-12T03:14:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.685+0000 7ff0de5d8700 1 -- 172.21.15.94:0/2932737389 >> v1:172.21.15.94:6800/1299282424 conn(0x7ff0c006ec40 legacy=0x7ff0c0071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.685+0000 7ff0de5d8700 1 -- 172.21.15.94:0/2932737389 >> v1:172.21.15.94:6790/0 conn(0x7ff0d8106010 legacy=0x7ff0d819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.685+0000 7ff0de5d8700 1 -- 172.21.15.94:0/2932737389 shutdown_connections 2024-03-12T03:14:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.685+0000 7ff0de5d8700 1 -- 172.21.15.94:0/2932737389 >> 172.21.15.94:0/2932737389 conn(0x7ff0d80fd180 msgr2=0x7ff0d8108460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.686+0000 7ff0de5d8700 1 -- 172.21.15.94:0/2932737389 shutdown_connections 2024-03-12T03:14:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:01.686+0000 7ff0de5d8700 1 -- 172.21.15.94:0/2932737389 wait complete. 2024-03-12T03:14:01.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:01 smithi151 ceph-mon[32923]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 2.3 KiB/s wr, 100 op/s 2024-03-12T03:14:02.405 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:14:02.405 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-12T03:14:02.405 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-12T03:14:02.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T03:14:02.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2932737389' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:14:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[29291]: from='client.34475 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:14:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[29291]: from='client.34475 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:14:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-12T03:14:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-03-12T03:14:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-12T03:14:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2932737389' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:14:02.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[40484]: from='client.34475 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:14:02.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[40484]: from='client.34475 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:14:02.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-12T03:14:02.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-03-12T03:14:02.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:02 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-12T03:14:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:02 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2932737389' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:14:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:02 smithi151 ceph-mon[32923]: from='client.34475 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:14:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:02 smithi151 ceph-mon[32923]: from='client.34475 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:14:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-12T03:14:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-03-12T03:14:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:02 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-12T03:14:03.286 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:14:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:03 smithi094 ceph-mon[29291]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 938 B/s wr, 25 op/s 2024-03-12T03:14:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:03 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-12T03:14:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:03 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-03-12T03:14:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:03 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-12T03:14:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:03 smithi094 ceph-mon[29291]: osdmap e70: 8 total, 8 up, 8 in 2024-03-12T03:14:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:03 smithi094 ceph-mon[40484]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 938 B/s wr, 25 op/s 2024-03-12T03:14:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:03 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-12T03:14:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:03 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-03-12T03:14:03.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:03 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-12T03:14:03.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:03 smithi094 ceph-mon[40484]: osdmap e70: 8 total, 8 up, 8 in 2024-03-12T03:14:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:03 smithi151 ceph-mon[32923]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 938 B/s wr, 25 op/s 2024-03-12T03:14:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:03 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-12T03:14:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:03 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-03-12T03:14:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:03 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-12T03:14:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:03 smithi151 ceph-mon[32923]: osdmap e70: 8 total, 8 up, 8 in 2024-03-12T03:14:04.872 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:04 smithi151 ceph-mon[32923]: osdmap e71: 8 total, 8 up, 8 in 2024-03-12T03:14:04.872 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:04 smithi151 ceph-mon[32923]: pgmap v212: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 7/690 objects degraded (1.014%) 2024-03-12T03:14:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:04 smithi094 ceph-mon[29291]: osdmap e71: 8 total, 8 up, 8 in 2024-03-12T03:14:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:04 smithi094 ceph-mon[29291]: pgmap v212: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 7/690 objects degraded (1.014%) 2024-03-12T03:14:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:04 smithi094 ceph-mon[40484]: osdmap e71: 8 total, 8 up, 8 in 2024-03-12T03:14:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:04 smithi094 ceph-mon[40484]: pgmap v212: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 7/690 objects degraded (1.014%) 2024-03-12T03:14:05.812 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:05 smithi151 ceph-mon[32923]: Health check failed: Degraded data redundancy: 7/690 objects degraded (1.014%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:14:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:05 smithi094 ceph-mon[29291]: Health check failed: Degraded data redundancy: 7/690 objects degraded (1.014%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:14:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:05 smithi094 ceph-mon[40484]: Health check failed: Degraded data redundancy: 7/690 objects degraded (1.014%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:14:06.826 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:06 smithi151 ceph-mon[32923]: pgmap v213: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 7/690 objects degraded (1.014%); 13 B/s, 1 objects/s recovering 2024-03-12T03:14:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:06 smithi094 ceph-mon[29291]: pgmap v213: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 7/690 objects degraded (1.014%); 13 B/s, 1 objects/s recovering 2024-03-12T03:14:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:06 smithi094 ceph-mon[40484]: pgmap v213: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 7/690 objects degraded (1.014%); 13 B/s, 1 objects/s recovering 2024-03-12T03:14:07.465 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:07 smithi151 systemd[1]: Starting Ceph prometheus.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:14:07.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.859+0000 7f22e6a89700 1 -- 172.21.15.151:0/1704311878 >> v1:172.21.15.151:6789/0 conn(0x7f22e0104f00 legacy=0x7f22e0107350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:07.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.859+0000 7f22e6a89700 1 -- 172.21.15.151:0/1704311878 shutdown_connections 2024-03-12T03:14:07.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.859+0000 7f22e6a89700 1 -- 172.21.15.151:0/1704311878 >> 172.21.15.151:0/1704311878 conn(0x7f22e00fc0b0 msgr2=0x7f22e00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:07.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.859+0000 7f22e6a89700 1 -- 172.21.15.151:0/1704311878 shutdown_connections 2024-03-12T03:14:07.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.859+0000 7f22e6a89700 1 -- 172.21.15.151:0/1704311878 wait complete. 2024-03-12T03:14:07.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.860+0000 7f22e6a89700 1 Processor -- start 2024-03-12T03:14:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.860+0000 7f22e6a89700 1 -- start start 2024-03-12T03:14:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22e6a89700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f22e019a230 con 0x7f22e0104f00 2024-03-12T03:14:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22e6a89700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f22e01a5860 con 0x7f22e0100ad0 2024-03-12T03:14:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22e6a89700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f22e01a6a40 con 0x7f22e0108c40 2024-03-12T03:14:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22e4825700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f22e0100ad0 0x7f22e0199700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:54272/0 (socket says 172.21.15.151:54272) 2024-03-12T03:14:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22e4825700 1 -- 172.21.15.151:0/1013751153 learned_addr learned my addr 172.21.15.151:0/1013751153 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-12T03:14:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677856450 0 0) 0x7f22e01a5860 con 0x7f22e0100ad0 2024-03-12T03:14:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22c80035b0 con 0x7f22e0100ad0 2024-03-12T03:14:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516937541 0 0) 0x7f22e019a230 con 0x7f22e0104f00 2024-03-12T03:14:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e01a5860 con 0x7f22e0104f00 2024-03-12T03:14:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735804425 0 0) 0x7f22e01a6a40 con 0x7f22e0108c40 2024-03-12T03:14:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e019a230 con 0x7f22e0108c40 2024-03-12T03:14:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3492937902 0 0) 0x7f22e01a5860 con 0x7f22e0104f00 2024-03-12T03:14:07.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22e01a6a40 con 0x7f22e0104f00 2024-03-12T03:14:07.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f22d4001430 con 0x7f22e0104f00 2024-03-12T03:14:07.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4197162949 0 0) 0x7f22e019a230 con 0x7f22e0108c40 2024-03-12T03:14:07.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22e01a5860 con 0x7f22e0108c40 2024-03-12T03:14:07.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.861+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2389656200 0 0) 0x7f22c80035b0 con 0x7f22e0100ad0 2024-03-12T03:14:07.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.862+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22e019a230 con 0x7f22e0100ad0 2024-03-12T03:14:07.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.862+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f22d0001830 con 0x7f22e0108c40 2024-03-12T03:14:07.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.862+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f22d8001bb0 con 0x7f22e0100ad0 2024-03-12T03:14:07.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.862+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3434548934 0 0) 0x7f22e01a6a40 con 0x7f22e0104f00 2024-03-12T03:14:07.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.862+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 >> v1:172.21.15.94:6790/0 conn(0x7f22e0108c40 legacy=0x7f22e01a3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:07.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.862+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 >> v1:172.21.15.151:6789/0 conn(0x7f22e0100ad0 legacy=0x7f22e0199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:07.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.862+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e01a7c20 con 0x7f22e0104f00 2024-03-12T03:14:07.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.862+0000 7f22e6a89700 1 -- 172.21.15.151:0/1013751153 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e01a48b0 con 0x7f22e0104f00 2024-03-12T03:14:07.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.862+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f22d4001b70 con 0x7f22e0104f00 2024-03-12T03:14:07.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.862+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f22d40050f0 con 0x7f22e0104f00 2024-03-12T03:14:07.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.862+0000 7f22e6a89700 1 -- 172.21.15.151:0/1013751153 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e01a4df0 con 0x7f22e0104f00 2024-03-12T03:14:07.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.863+0000 7f22e6a89700 1 -- 172.21.15.151:0/1013751153 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e010e650 con 0x7f22e0104f00 2024-03-12T03:14:07.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.864+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 16) v1 ==== 91770+0+0 (unknown 3875668065 0 0) 0x7f22d4003ab0 con 0x7f22e0104f00 2024-03-12T03:14:07.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.864+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6001+0+0 (unknown 1636461450 0 0) 0x7f22d4088160 con 0x7f22e0104f00 2024-03-12T03:14:07.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:07.869+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f22d40050d0 con 0x7f22e0104f00 2024-03-12T03:14:08.121 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:07 smithi151 podman[61580]: 2024-03-12 03:14:07.891762101 +0000 UTC m=+0.018686872 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-12T03:14:08.121 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:07 smithi151 podman[61580]: 2024-03-12 03:14:07.99937723 +0000 UTC m=+0.126302000 container create 421b13c53e55401f2174f7f6b40f9e72adb55fe6d131f1eb88b8077a124f9b0f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:14:08.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:08.149+0000 7f22e6a89700 1 -- 172.21.15.151:0/1013751153 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f22e010dd50 con 0x7f22e0104f00 2024-03-12T03:14:08.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:08.157+0000 7f22ddffb700 1 -- 172.21.15.151:0/1013751153 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) v1 ==== 170+0+59 (unknown 3508017507 0 2052087441) 0x7f22d4055940 con 0x7f22e0104f00 2024-03-12T03:14:08.160 INFO:teuthology.orchestra.run.smithi151.stdout:[client.1] 2024-03-12T03:14:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCAyO9lHMAECRAA6BghpMR6h4cnIKDO6Ct5/A== 2024-03-12T03:14:08.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:08.159+0000 7f22e6a89700 1 -- 172.21.15.151:0/1013751153 >> v1:172.21.15.94:6800/1299282424 conn(0x7f22c806ef30 legacy=0x7f22c80713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:08.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:08.159+0000 7f22e6a89700 1 -- 172.21.15.151:0/1013751153 >> v1:172.21.15.94:6789/0 conn(0x7f22e0104f00 legacy=0x7f22e019fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:08.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:08.159+0000 7f22e6a89700 1 -- 172.21.15.151:0/1013751153 shutdown_connections 2024-03-12T03:14:08.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:08.159+0000 7f22e6a89700 1 -- 172.21.15.151:0/1013751153 >> 172.21.15.151:0/1013751153 conn(0x7f22e00fc0b0 msgr2=0x7f22e006a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:08.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:08.159+0000 7f22e6a89700 1 -- 172.21.15.151:0/1013751153 shutdown_connections 2024-03-12T03:14:08.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-12T03:14:08.159+0000 7f22e6a89700 1 -- 172.21.15.151:0/1013751153 wait complete. 2024-03-12T03:14:08.402 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/1013751153' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:14:08.403 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.151:0/1013751153' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:14:08.403 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 podman[61580]: 2024-03-12 03:14:08.277071039 +0000 UTC m=+0.403995811 container init 421b13c53e55401f2174f7f6b40f9e72adb55fe6d131f1eb88b8077a124f9b0f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:14:08.403 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 podman[61580]: 2024-03-12 03:14:08.279937339 +0000 UTC m=+0.406862118 container start 421b13c53e55401f2174f7f6b40f9e72adb55fe6d131f1eb88b8077a124f9b0f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:14:08.403 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.302Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-12T03:14:08.403 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.302Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.302Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi151 (none))" 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.303Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.303Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.304Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.304Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.305Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.305Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.306Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.306Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.492µs 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.306Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.307Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.307Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=15.385µs wal_replay_duration=1.422897ms wbl_replay_duration=86ns total_replay_duration=1.454858ms 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.308Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-12T03:14:08.404 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.308Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-12T03:14:08.405 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.308Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-12T03:14:08.405 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.328Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.319528ms db_storage=763ns remote_storage=1.006µs web_handler=378ns query_engine=552ns scrape=424.27µs scrape_sd=90.357µs notify=666ns notify_sd=1.229µs rules=19.558351ms tracing=6.024µs 2024-03-12T03:14:08.405 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.329Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-12T03:14:08.405 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:14:08.329Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-12T03:14:08.405 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 bash[61580]: 421b13c53e55401f2174f7f6b40f9e72adb55fe6d131f1eb88b8077a124f9b0f 2024-03-12T03:14:08.405 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:14:08 smithi151 systemd[1]: Started Ceph prometheus.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:14:08.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/1013751153' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:14:08.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.151:0/1013751153' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:14:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/1013751153' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:14:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.151:0/1013751153' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:14:08.951 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-12T03:14:08.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-12T03:14:08.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-12T03:14:08.995 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-12T03:14:08.995 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-12T03:14:08.995 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-12T03:14:09.245 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:09.694 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:09 smithi094 ceph-mon[29291]: pgmap v214: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 7/690 objects degraded (1.014%); 41 B/s, 1 objects/s recovering 2024-03-12T03:14:09.695 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:09 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:14:09.695 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:09 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:14:09.695 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:09 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:14:09.695 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:09 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T03:14:09.695 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:09 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ignoring --setuser ceph since I am not root 2024-03-12T03:14:09.695 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:09 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ignoring --setgroup ceph since I am not root 2024-03-12T03:14:09.696 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:09 smithi094 ceph-mon[40484]: pgmap v214: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 7/690 objects degraded (1.014%); 41 B/s, 1 objects/s recovering 2024-03-12T03:14:09.696 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:09 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:14:09.696 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:09 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:14:09.696 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:09 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:14:09.696 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:09 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T03:14:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:09 smithi151 ceph-mon[32923]: pgmap v214: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 7/690 objects degraded (1.014%); 41 B/s, 1 objects/s recovering 2024-03-12T03:14:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:09 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:14:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:09 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:14:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:09 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' 2024-03-12T03:14:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:09 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T03:14:09.715 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: ignoring --setuser ceph since I am not root 2024-03-12T03:14:09.715 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: ignoring --setgroup ceph since I am not root 2024-03-12T03:14:10.394 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:10.035+0000 7fd9e9e6b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:14:10.395 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:10.128+0000 7fd9e9e6b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:14:10.464 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:10 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:10.020+0000 7f67160f7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:14:10.464 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:10 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:10.113+0000 7f67160f7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:14:10.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:10 smithi094 ceph-mon[29291]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T03:14:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:10 smithi094 ceph-mon[29291]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-12T03:14:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:10 smithi094 ceph-mon[40484]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T03:14:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:10 smithi094 ceph-mon[40484]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-12T03:14:10.753 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:10 smithi151 ceph-mon[32923]: from='mgr.14152 v1:172.21.15.94:0/1155326693' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T03:14:10.754 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:10 smithi151 ceph-mon[32923]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-12T03:14:10.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.946+0000 7fc6f6f62700 1 -- 172.21.15.94:0/3480373304 >> v1:172.21.15.94:6790/0 conn(0x7fc6f0109920 legacy=0x7fc6f010bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:10.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.946+0000 7fc6f6f62700 1 -- 172.21.15.94:0/3480373304 shutdown_connections 2024-03-12T03:14:10.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.946+0000 7fc6f6f62700 1 -- 172.21.15.94:0/3480373304 >> 172.21.15.94:0/3480373304 conn(0x7fc6f00fc580 msgr2=0x7fc6f00fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:10.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.946+0000 7fc6f6f62700 1 -- 172.21.15.94:0/3480373304 shutdown_connections 2024-03-12T03:14:10.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.946+0000 7fc6f6f62700 1 -- 172.21.15.94:0/3480373304 wait complete. 2024-03-12T03:14:10.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.947+0000 7fc6f6f62700 1 Processor -- start 2024-03-12T03:14:10.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.947+0000 7fc6f6f62700 1 -- start start 2024-03-12T03:14:10.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.947+0000 7fc6f6f62700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6f01a5ab0 con 0x7fc6f0109920 2024-03-12T03:14:10.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.947+0000 7fc6f6f62700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6f01a6cf0 con 0x7fc6f010d4f0 2024-03-12T03:14:10.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.947+0000 7fc6f6f62700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6f01a7f30 con 0x7fc6f0106a70 2024-03-12T03:14:10.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.947+0000 7fc6effff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc6f0109920 0x7fc6f01a09a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45730/0 (socket says 172.21.15.94:45730) 2024-03-12T03:14:10.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.947+0000 7fc6f4cfe700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fc6f0106a70 0x7fc6f010cbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:58352/0 (socket says 172.21.15.94:58352) 2024-03-12T03:14:10.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.947+0000 7fc6effff700 1 -- 172.21.15.94:0/2993232487 learned_addr learned my addr 172.21.15.94:0/2993232487 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:10.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998608193 0 0) 0x7fc6f01a5ab0 con 0x7fc6f0109920 2024-03-12T03:14:10.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e00035b0 con 0x7fc6f0109920 2024-03-12T03:14:10.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946595779 0 0) 0x7fc6f01a7f30 con 0x7fc6f0106a70 2024-03-12T03:14:10.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f01a5ab0 con 0x7fc6f0106a70 2024-03-12T03:14:10.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429668334 0 0) 0x7fc6f01a6cf0 con 0x7fc6f010d4f0 2024-03-12T03:14:10.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f01a7f30 con 0x7fc6f010d4f0 2024-03-12T03:14:10.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1859039215 0 0) 0x7fc6e00035b0 con 0x7fc6f0109920 2024-03-12T03:14:10.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6f01a6cf0 con 0x7fc6f0109920 2024-03-12T03:14:10.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc6dc001510 con 0x7fc6f0109920 2024-03-12T03:14:10.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 343279981 0 0) 0x7fc6f01a7f30 con 0x7fc6f010d4f0 2024-03-12T03:14:10.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6e00035b0 con 0x7fc6f010d4f0 2024-03-12T03:14:10.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3354738489 0 0) 0x7fc6f01a5ab0 con 0x7fc6f0106a70 2024-03-12T03:14:10.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6f01a7f30 con 0x7fc6f0106a70 2024-03-12T03:14:10.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc6e8001bb0 con 0x7fc6f010d4f0 2024-03-12T03:14:10.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc6e4001870 con 0x7fc6f0106a70 2024-03-12T03:14:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2699329574 0 0) 0x7fc6f01a6cf0 con 0x7fc6f0109920 2024-03-12T03:14:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 >> v1:172.21.15.94:6790/0 conn(0x7fc6f0106a70 legacy=0x7fc6f010cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 >> v1:172.21.15.151:6789/0 conn(0x7fc6f010d4f0 legacy=0x7fc6f01a41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f01a9170 con 0x7fc6f0109920 2024-03-12T03:14:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6f6f62700 1 -- 172.21.15.94:0/2993232487 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f01a8160 con 0x7fc6f0109920 2024-03-12T03:14:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fc6dc001af0 con 0x7fc6f0109920 2024-03-12T03:14:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc6dc004fc0 con 0x7fc6f0109920 2024-03-12T03:14:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.948+0000 7fc6f6f62700 1 -- 172.21.15.94:0/2993232487 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f01a8620 con 0x7fc6f0109920 2024-03-12T03:14:10.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.950+0000 7fc6f6f62700 1 -- 172.21.15.94:0/2993232487 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f01021b0 con 0x7fc6f0109920 2024-03-12T03:14:10.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.950+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 17) v1 ==== 91784+0+0 (unknown 2253508574 0 0) 0x7fc6dc004fa0 con 0x7fc6f0109920 2024-03-12T03:14:10.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.950+0000 7fc6f4cfe700 1 -- 172.21.15.94:0/2993232487 >> v1:172.21.15.94:6800/1299282424 conn(0x7fc6e006ed50 legacy=0x7fc6e0071210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/1299282424 2024-03-12T03:14:10.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.950+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6001+0+0 (unknown 1636461450 0 0) 0x7fc6dc0892c0 con 0x7fc6f0109920 2024-03-12T03:14:10.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:10.955+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fc6dc056aa0 con 0x7fc6f0109920 2024-03-12T03:14:11.046 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:10.786+0000 7fd9e9e6b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:14:11.046 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:10.882+0000 7fd9e9e6b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:14:11.046 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:11 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:11.043+0000 7fd9e9e6b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:14:11.083 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:10 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:10.751+0000 7f67160f7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:14:11.084 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:10 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:10.843+0000 7f67160f7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:14:11.084 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:10 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:10.986+0000 7f67160f7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:14:11.084 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:11 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:11.081+0000 7f67160f7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:14:11.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:11.151+0000 7fc6f4cfe700 1 -- 172.21.15.94:0/2993232487 >> v1:172.21.15.94:6800/1299282424 conn(0x7fc6e006ed50 legacy=0x7fc6e0071210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/1299282424 2024-03-12T03:14:11.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:11.234+0000 7fc6f6f62700 1 -- 172.21.15.94:0/2993232487 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fc6f010b210 con 0x7fc6f0109920 2024-03-12T03:14:11.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:11.237+0000 7fc6edffb700 1 -- 172.21.15.94:0/2993232487 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+175597 (unknown 831476310 0 2846027078) 0x7fc6dc001ca0 con 0x7fc6f0109920 2024-03-12T03:14:11.242 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:14:11.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:11.240+0000 7fc6f6f62700 1 -- 172.21.15.94:0/2993232487 >> v1:172.21.15.94:6800/1299282424 conn(0x7fc6e006ed50 legacy=0x7fc6e0071210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T03:14:11.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:11.240+0000 7fc6f6f62700 1 -- 172.21.15.94:0/2993232487 >> v1:172.21.15.94:6789/0 conn(0x7fc6f0109920 legacy=0x7fc6f01a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:11.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:11.240+0000 7fc6f6f62700 1 -- 172.21.15.94:0/2993232487 shutdown_connections 2024-03-12T03:14:11.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:11.240+0000 7fc6f6f62700 1 -- 172.21.15.94:0/2993232487 >> 172.21.15.94:0/2993232487 conn(0x7fc6f00fc580 msgr2=0x7fc6f0110160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:11.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:11.240+0000 7fc6f6f62700 1 -- 172.21.15.94:0/2993232487 shutdown_connections 2024-03-12T03:14:11.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:11.240+0000 7fc6f6f62700 1 -- 172.21.15.94:0/2993232487 wait complete. 2024-03-12T03:14:11.430 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:11 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:11.165+0000 7f67160f7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:14:11.459 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:11 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:11.141+0000 7fd9e9e6b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:14:11.459 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:11 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:11.226+0000 7fd9e9e6b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:14:11.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2993232487' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:14:11.715 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:11 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:11.428+0000 7f67160f7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:14:11.715 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:11 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:11.538+0000 7f67160f7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:14:11.715 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:11 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:11.623+0000 7f67160f7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:14:11.792 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2993232487' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:14:11.792 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2993232487' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:14:11.792 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:11 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:11.501+0000 7fd9e9e6b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:14:11.793 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:11 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:11.609+0000 7fd9e9e6b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:14:11.793 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:11 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:11.695+0000 7fd9e9e6b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:14:11.985 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6800","nonce":1299282424}]},"active_addr":"172.21.15.94:6800/1299282424","active_change":"2024-03-12T03:09:02.315158+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.94:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":1893861386}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":1936764519}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":2533680730}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":1656942945}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":1546784249}]}]} 2024-03-12T03:14:11.989 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-12T03:14:11.989 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-12T03:14:11.989 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:14:12.209 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:11 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:11.789+0000 7fd9e9e6b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:14:12.210 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:11 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:11.968+0000 7fd9e9e6b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:14:12.214 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:11 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:11.717+0000 7f67160f7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:14:12.214 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:11 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:11.893+0000 7f67160f7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:14:12.229 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:13.116 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:12.852+0000 7f67160f7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:14:13.116 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:12.983+0000 7f67160f7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:14:13.397 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:13.328+0000 7f67160f7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:14:13.438 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:12 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:12.965+0000 7fd9e9e6b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:14:13.438 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:13 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:13.102+0000 7fd9e9e6b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:14:13.438 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:13 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:13.433+0000 7fd9e9e6b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:14:13.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:13 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:13.532+0000 7fd9e9e6b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:14:13.714 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:13.425+0000 7f67160f7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:14:13.714 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:13.634+0000 7f67160f7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:14:14.011 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:13 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:13.733+0000 7fd9e9e6b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:14:14.011 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:13 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:13.926+0000 7fd9e9e6b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:14:14.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.028+0000 7fa244b39700 1 -- 172.21.15.94:0/135500921 >> v1:172.21.15.94:6789/0 conn(0x7fa240109920 legacy=0x7fa24010bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:14.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.028+0000 7fa244b39700 1 -- 172.21.15.94:0/135500921 shutdown_connections 2024-03-12T03:14:14.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.028+0000 7fa244b39700 1 -- 172.21.15.94:0/135500921 >> 172.21.15.94:0/135500921 conn(0x7fa2400fc580 msgr2=0x7fa2400fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:14.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.029+0000 7fa244b39700 1 -- 172.21.15.94:0/135500921 shutdown_connections 2024-03-12T03:14:14.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.029+0000 7fa244b39700 1 -- 172.21.15.94:0/135500921 wait complete. 2024-03-12T03:14:14.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.029+0000 7fa244b39700 1 Processor -- start 2024-03-12T03:14:14.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.029+0000 7fa244b39700 1 -- start start 2024-03-12T03:14:14.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa244b39700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2401a9010 con 0x7fa240109920 2024-03-12T03:14:14.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa244b39700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2401aa230 con 0x7fa24010d4f0 2024-03-12T03:14:14.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa244b39700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2401ab470 con 0x7fa240106a70 2024-03-12T03:14:14.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa23e59c700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fa240106a70 0x7fa24010cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:58374/0 (socket says 172.21.15.94:58374) 2024-03-12T03:14:14.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa23e59c700 1 -- 172.21.15.94:0/3202873580 learned_addr learned my addr 172.21.15.94:0/3202873580 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:14.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259579586 0 0) 0x7fa2401ab470 con 0x7fa240106a70 2024-03-12T03:14:14.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2280035b0 con 0x7fa240106a70 2024-03-12T03:14:14.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155853151 0 0) 0x7fa2401a9010 con 0x7fa240109920 2024-03-12T03:14:14.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2401ab470 con 0x7fa240109920 2024-03-12T03:14:14.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426688247 0 0) 0x7fa2401aa230 con 0x7fa24010d4f0 2024-03-12T03:14:14.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2401a9010 con 0x7fa24010d4f0 2024-03-12T03:14:14.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3653898004 0 0) 0x7fa2280035b0 con 0x7fa240106a70 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2401aa230 con 0x7fa240106a70 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fa230001930 con 0x7fa240106a70 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2106079283 0 0) 0x7fa2401ab470 con 0x7fa240109920 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2280035b0 con 0x7fa240109920 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2030091119 0 0) 0x7fa2401a9010 con 0x7fa24010d4f0 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2401ab470 con 0x7fa24010d4f0 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fa234001430 con 0x7fa240109920 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.030+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fa238001bb0 con 0x7fa24010d4f0 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.031+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2731639162 0 0) 0x7fa2401aa230 con 0x7fa240106a70 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.031+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 >> v1:172.21.15.151:6789/0 conn(0x7fa24010d4f0 legacy=0x7fa2401a76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.031+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 >> v1:172.21.15.94:6789/0 conn(0x7fa240109920 legacy=0x7fa2401a3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.031+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2401ac6b0 con 0x7fa240106a70 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.031+0000 7fa244b39700 1 -- 172.21.15.94:0/3202873580 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2401a9240 con 0x7fa240106a70 2024-03-12T03:14:14.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.031+0000 7fa244b39700 1 -- 172.21.15.94:0/3202873580 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2401a9700 con 0x7fa240106a70 2024-03-12T03:14:14.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.031+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fa230003b30 con 0x7fa240106a70 2024-03-12T03:14:14.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.031+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fa230005cc0 con 0x7fa240106a70 2024-03-12T03:14:14.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.032+0000 7fa244b39700 1 -- 172.21.15.94:0/3202873580 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa220005120 con 0x7fa240106a70 2024-03-12T03:14:14.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.032+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 17) v1 ==== 91784+0+0 (unknown 2253508574 0 0) 0x7fa2300036a0 con 0x7fa240106a70 2024-03-12T03:14:14.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.033+0000 7fa23dd9b700 1 -- 172.21.15.94:0/3202873580 >> v1:172.21.15.94:6800/1299282424 conn(0x7fa22806ecf0 legacy=0x7fa2280711b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/1299282424 2024-03-12T03:14:14.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.033+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6001+0+0 (unknown 1636461450 0 0) 0x7fa2300890b0 con 0x7fa240106a70 2024-03-12T03:14:14.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.037+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fa230005ca0 con 0x7fa240106a70 2024-03-12T03:14:14.129 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:13.829+0000 7f67160f7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:14:14.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.234+0000 7fa23dd9b700 1 -- 172.21.15.94:0/3202873580 >> v1:172.21.15.94:6800/1299282424 conn(0x7fa22806ecf0 legacy=0x7fa2280711b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6800/1299282424 2024-03-12T03:14:14.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.238+0000 7fa244b39700 1 -- 172.21.15.94:0/3202873580 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa220005410 con 0x7fa240106a70 2024-03-12T03:14:14.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.240+0000 7fa22f7fe700 1 -- 172.21.15.94:0/3202873580 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+20676 (unknown 2035373579 0 2267061892) 0x7fa230056950 con 0x7fa240106a70 2024-03-12T03:14:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:14:14.244 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":71,"fsid":"964142e8-e01d-11ee-95c8-87774f69a715","created":"2024-03-12T03:08:07.892137+0000","modified":"2024-03-12T03:14:03.687356+0000","last_up_change":"2024-03-12T03:13:24.457659+0000","last_in_change":"2024-03-12T03:13:07.180424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:11:20.645672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-12T03:13:36.816579+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-12T03:13:37.458899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-12T03:13:39.236115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-12T03:13:41.174045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-12T03:13:43.233964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"003c53a3-baa3-4abc-bbed-e8a0c0db7ab5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":2935792501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":2935792501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":2935792501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":2935792501}]},"public_addr":"172.21.15.94:6801/2935792501","cluster_addr":"172.21.15.94:6802/2935792501","heartbeat_back_addr":"172.21.15.94:6804/2935792501","heartbeat_front_addr":"172.21.15.94:6803/2935792501","state":["exists","up"]},{"osd":1,"uuid":"91c8e6b7-d57f-492b-bc40-9f46769a1c0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":1056655004}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":1056655004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":1056655004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":1056655004}]},"public_addr":"172.21.15.94:6805/1056655004","cluster_addr":"172.21.15.94:6806/1056655004","heartbeat_back_addr":"172.21.15.94:6808/1056655004","heartbeat_front_addr":"172.21.15.94:6807/1056655004","state":["exists","up"]},{"osd":2,"uuid":"4b0ab939-b3de-4821-9c65-348f5172981d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":2929979688}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":2929979688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":2929979688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":2929979688}]},"public_addr":"172.21.15.94:6809/2929979688","cluster_addr":"172.21.15.94:6810/2929979688","heartbeat_back_addr":"172.21.15.94:6812/2929979688","heartbeat_front_addr":"172.21.15.94:6811/2929979688","state":["exists","up"]},{"osd":3,"uuid":"3fd5c9c8-6304-4233-a4df-8c4c4bda5f79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":4235334318}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":4235334318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6816","nonce":4235334318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":4235334318}]},"public_addr":"172.21.15.94:6813/4235334318","cluster_addr":"172.21.15.94:6814/4235334318","heartbeat_back_addr":"172.21.15.94:6816/4235334318","heartbeat_front_addr":"172.21.15.94:6815/4235334318","state":["exists","up"]},{"osd":4,"uuid":"93461da9-bc47-489d-835e-89fcf88c9287","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6800","nonce":2666240674}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6801","nonce":2666240674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6803","nonce":2666240674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6802","nonce":2666240674}]},"public_addr":"172.21.15.151:6800/2666240674","cluster_addr":"172.21.15.151:6801/2666240674","heartbeat_back_addr":"172.21.15.151:6803/2666240674","heartbeat_front_addr":"172.21.15.151:6802/2666240674","state":["exists","up"]},{"osd":5,"uuid":"4c72b1d7-28f6-4ee3-bb80-57ff924abc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6804","nonce":2718612978}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6805","nonce":2718612978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6807","nonce":2718612978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6806","nonce":2718612978}]},"public_addr":"172.21.15.151:6804/2718612978","cluster_addr":"172.21.15.151:6805/2718612978","heartbeat_back_addr":"172.21.15.151:6807/2718612978","heartbeat_front_addr":"172.21.15.151:6806/2718612978","state":["exists","up"]},{"osd":6,"uuid":"d852205d-6b9d-4a4f-83b4-a0e59aa37657","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6808","nonce":1497503104}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6809","nonce":1497503104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6811","nonce":1497503104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6810","nonce":1497503104}]},"public_addr":"172.21.15.151:6808/1497503104","cluster_addr":"172.21.15.151:6809/1497503104","heartbeat_back_addr":"172.21.15.151:6811/1497503104","heartbeat_front_addr":"172.21.15.151:6810/1497503104","state":["exists","up"]},{"osd":7,"uuid":"cd9fca37-a6f2-46fc-90de-cd3790ad3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6812","nonce":683127336}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6813","nonce":683127336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6815","nonce":683127336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6814","nonce":683127336}]},"public_addr":"172.21.15.151:6812/683127336","cluster_addr":"172.21.15.151:6813/683127336","heartbeat_back_addr":"172.21.15.151:6815/683127336","heartbeat_front_addr":"172.21.15.151:6814/683127336","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:10:26.284321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:10:52.628310+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:11:18.278275+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:11:44.195150+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:12:07.431888+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:12:32.407393+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:12:57.133251+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:13:23.236080+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/2796836265":"2024-03-13T03:09:02.315045+0000","172.21.15.94:0/1664309417":"2024-03-13T03:08:34.850802+0000","172.21.15.94:0/3802544616":"2024-03-13T03:08:34.850802+0000","172.21.15.94:6800/3548280253":"2024-03-13T03:09:02.315045+0000","172.21.15.94:6800/2955156665":"2024-03-13T03:08:34.850802+0000","172.21.15.94:0/782403058":"2024-03-13T03:08:34.850802+0000","172.21.15.94:0/3132181":"2024-03-13T03:09:02.315045+0000","172.21.15.94:0/3815053600":"2024-03-13T03:09:02.315045+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:14:14.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.242+0000 7fa244b39700 1 -- 172.21.15.94:0/3202873580 >> v1:172.21.15.94:6800/1299282424 conn(0x7fa22806ecf0 legacy=0x7fa2280711b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T03:14:14.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.242+0000 7fa244b39700 1 -- 172.21.15.94:0/3202873580 >> v1:172.21.15.94:6790/0 conn(0x7fa240106a70 legacy=0x7fa24010cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:14.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.242+0000 7fa244b39700 1 -- 172.21.15.94:0/3202873580 shutdown_connections 2024-03-12T03:14:14.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.242+0000 7fa244b39700 1 -- 172.21.15.94:0/3202873580 >> 172.21.15.94:0/3202873580 conn(0x7fa2400fc580 msgr2=0x7fa2400fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:14.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.242+0000 7fa244b39700 1 -- 172.21.15.94:0/3202873580 shutdown_connections 2024-03-12T03:14:14.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:14.242+0000 7fa244b39700 1 -- 172.21.15.94:0/3202873580 wait complete. 2024-03-12T03:14:14.294 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:14 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:14.229+0000 7fd9e9e6b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:14:14.402 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3202873580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:14:14.403 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:14.127+0000 7f67160f7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:14:14.616 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3202873580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:14:14.616 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3202873580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:14:14.617 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:14 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:14.493+0000 7fd9e9e6b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:14:14.714 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:14.400+0000 7f67160f7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:14:14.714 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:14.638+0000 7f67160f7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:14:14.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:14 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:14.736+0000 7fd9e9e6b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:14:15.078 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-12T03:14:15.078 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:14:15.324 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:15.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:15 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:15.584+0000 7fd9e9e6b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:14:15.714 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:15 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:15.438+0000 7f67160f7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:14:16.324 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:16.007+0000 7f67160f7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:14:16.325 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:16.106+0000 7f67160f7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:14:16.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-mon[29291]: Standby manager daemon x restarted 2024-03-12T03:14:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-mon[29291]: Standby manager daemon x started 2024-03-12T03:14:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-mon[29291]: from='mgr.? v1:172.21.15.151:0/119715558' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:14:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-mon[29291]: from='mgr.? v1:172.21.15.151:0/119715558' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:14:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-mon[29291]: from='mgr.? v1:172.21.15.151:0/119715558' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:14:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-mon[29291]: from='mgr.? v1:172.21.15.151:0/119715558' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:14:16.460 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:16.163+0000 7fd9e9e6b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:14:16.460 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:16.262+0000 7fd9e9e6b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:14:16.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-mon[40484]: Standby manager daemon x restarted 2024-03-12T03:14:16.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-mon[40484]: Standby manager daemon x started 2024-03-12T03:14:16.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-mon[40484]: from='mgr.? v1:172.21.15.151:0/119715558' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:14:16.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-mon[40484]: from='mgr.? v1:172.21.15.151:0/119715558' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:14:16.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-mon[40484]: from='mgr.? v1:172.21.15.151:0/119715558' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:14:16.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-mon[40484]: from='mgr.? v1:172.21.15.151:0/119715558' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:14:16.629 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-mon[32923]: Standby manager daemon x restarted 2024-03-12T03:14:16.629 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-mon[32923]: Standby manager daemon x started 2024-03-12T03:14:16.629 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-mon[32923]: from='mgr.? v1:172.21.15.151:0/119715558' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:14:16.630 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-mon[32923]: from='mgr.? v1:172.21.15.151:0/119715558' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:14:16.630 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-mon[32923]: from='mgr.? v1:172.21.15.151:0/119715558' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:14:16.630 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-mon[32923]: from='mgr.? v1:172.21.15.151:0/119715558' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:14:16.630 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:16.322+0000 7f67160f7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:14:16.630 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: [12/Mar/2024:03:14:16] ENGINE Bus STARTING 2024-03-12T03:14:16.630 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: CherryPy Checker: 2024-03-12T03:14:16.630 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: The Application mounted at '' has an empty config. 2024-03-12T03:14:16.630 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: 2024-03-12T03:14:16.630 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: [12/Mar/2024:03:14:16] ENGINE Serving on http://:::9283 2024-03-12T03:14:16.630 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:14:16 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x[34769]: [12/Mar/2024:03:14:16] ENGINE Bus STARTED 2024-03-12T03:14:16.745 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:16.489+0000 7fd9e9e6b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:14:16.746 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:14:16] ENGINE Bus STARTING 2024-03-12T03:14:16.746 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: CherryPy Checker: 2024-03-12T03:14:16.746 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: The Application mounted at '' has an empty config. 2024-03-12T03:14:16.746 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:14:16.746 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:14:16] ENGINE Serving on http://:::9283 2024-03-12T03:14:16.746 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:14:16] ENGINE Bus STARTED 2024-03-12T03:14:17.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.243+0000 7f7e2e848700 1 -- 172.21.15.94:0/1951451092 >> v1:172.21.15.94:6789/0 conn(0x7f7e2810ab90 legacy=0x7f7e2810cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:17.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.243+0000 7f7e2e848700 1 -- 172.21.15.94:0/1951451092 shutdown_connections 2024-03-12T03:14:17.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.243+0000 7f7e2e848700 1 -- 172.21.15.94:0/1951451092 >> 172.21.15.94:0/1951451092 conn(0x7f7e280fc5d0 msgr2=0x7f7e280fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:17.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.244+0000 7f7e2e848700 1 -- 172.21.15.94:0/1951451092 shutdown_connections 2024-03-12T03:14:17.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.244+0000 7f7e2e848700 1 -- 172.21.15.94:0/1951451092 wait complete. 2024-03-12T03:14:17.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.244+0000 7f7e2e848700 1 Processor -- start 2024-03-12T03:14:17.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.244+0000 7f7e2e848700 1 -- start start 2024-03-12T03:14:17.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e2e848700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7e281a8f40 con 0x7f7e2810e6a0 2024-03-12T03:14:17.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e2e848700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7e281aa160 con 0x7f7e28102d80 2024-03-12T03:14:17.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e2e848700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7e281ab3a0 con 0x7f7e2810ab90 2024-03-12T03:14:17.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e2cde5700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7e2810e6a0 0x7f7e281a7600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43298/0 (socket says 172.21.15.94:43298) 2024-03-12T03:14:17.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e2cde5700 1 -- 172.21.15.94:0/3307199808 learned_addr learned my addr 172.21.15.94:0/3307199808 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:17.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e277fe700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f7e2810ab90 0x7f7e281a3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:57392/0 (socket says 172.21.15.94:57392) 2024-03-12T03:14:17.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093645695 0 0) 0x7f7e281a8f40 con 0x7f7e2810e6a0 2024-03-12T03:14:17.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e100035b0 con 0x7f7e2810e6a0 2024-03-12T03:14:17.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703670866 0 0) 0x7f7e281aa160 con 0x7f7e28102d80 2024-03-12T03:14:17.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e281a8f40 con 0x7f7e28102d80 2024-03-12T03:14:17.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1204950079 0 0) 0x7f7e281a8f40 con 0x7f7e28102d80 2024-03-12T03:14:17.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e281aa160 con 0x7f7e28102d80 2024-03-12T03:14:17.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1016135477 0 0) 0x7f7e100035b0 con 0x7f7e2810e6a0 2024-03-12T03:14:17.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e281a8f40 con 0x7f7e2810e6a0 2024-03-12T03:14:17.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.245+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f7e180018b0 con 0x7f7e28102d80 2024-03-12T03:14:17.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.246+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f7e200014b0 con 0x7f7e2810e6a0 2024-03-12T03:14:17.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.246+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 270977205 0 0) 0x7f7e281aa160 con 0x7f7e28102d80 2024-03-12T03:14:17.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.246+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 >> v1:172.21.15.94:6790/0 conn(0x7f7e2810ab90 legacy=0x7f7e281a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:17.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.246+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 >> v1:172.21.15.94:6789/0 conn(0x7f7e2810e6a0 legacy=0x7f7e281a7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:17.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.246+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e281ac5e0 con 0x7f7e28102d80 2024-03-12T03:14:17.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.246+0000 7f7e2e848700 1 -- 172.21.15.94:0/3307199808 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e281aa390 con 0x7f7e28102d80 2024-03-12T03:14:17.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.246+0000 7f7e2e848700 1 -- 172.21.15.94:0/3307199808 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e281aa850 con 0x7f7e28102d80 2024-03-12T03:14:17.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.246+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f7e18003b10 con 0x7f7e28102d80 2024-03-12T03:14:17.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.246+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f7e18004aa0 con 0x7f7e28102d80 2024-03-12T03:14:17.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.247+0000 7f7e2e848700 1 -- 172.21.15.94:0/3307199808 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e14005120 con 0x7f7e28102d80 2024-03-12T03:14:17.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.248+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 19) v1 ==== 91436+0+0 (unknown 184541572 0 0) 0x7f7e1801b180 con 0x7f7e28102d80 2024-03-12T03:14:17.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.249+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7f7e180879d0 con 0x7f7e28102d80 2024-03-12T03:14:17.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.258+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f7e18055610 con 0x7f7e28102d80 2024-03-12T03:14:17.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.472+0000 7f7e2e848700 1 -- 172.21.15.94:0/3307199808 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e14005410 con 0x7f7e28102d80 2024-03-12T03:14:17.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.474+0000 7f7e257fa700 1 -- 172.21.15.94:0/3307199808 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+21051 (unknown 1686360691 0 206174209) 0x7f7e18054f10 con 0x7f7e28102d80 2024-03-12T03:14:17.477 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:14:17.477 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":72,"fsid":"964142e8-e01d-11ee-95c8-87774f69a715","created":"2024-03-12T03:08:07.892137+0000","modified":"2024-03-12T03:14:16.504482+0000","last_up_change":"2024-03-12T03:13:24.457659+0000","last_in_change":"2024-03-12T03:13:07.180424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:11:20.645672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-12T03:13:36.816579+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-12T03:13:37.458899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-12T03:13:39.236115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-12T03:13:41.174045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-12T03:13:43.233964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"003c53a3-baa3-4abc-bbed-e8a0c0db7ab5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":2935792501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":2935792501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":2935792501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":2935792501}]},"public_addr":"172.21.15.94:6801/2935792501","cluster_addr":"172.21.15.94:6802/2935792501","heartbeat_back_addr":"172.21.15.94:6804/2935792501","heartbeat_front_addr":"172.21.15.94:6803/2935792501","state":["exists","up"]},{"osd":1,"uuid":"91c8e6b7-d57f-492b-bc40-9f46769a1c0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":1056655004}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":1056655004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":1056655004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":1056655004}]},"public_addr":"172.21.15.94:6805/1056655004","cluster_addr":"172.21.15.94:6806/1056655004","heartbeat_back_addr":"172.21.15.94:6808/1056655004","heartbeat_front_addr":"172.21.15.94:6807/1056655004","state":["exists","up"]},{"osd":2,"uuid":"4b0ab939-b3de-4821-9c65-348f5172981d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":2929979688}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":2929979688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":2929979688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":2929979688}]},"public_addr":"172.21.15.94:6809/2929979688","cluster_addr":"172.21.15.94:6810/2929979688","heartbeat_back_addr":"172.21.15.94:6812/2929979688","heartbeat_front_addr":"172.21.15.94:6811/2929979688","state":["exists","up"]},{"osd":3,"uuid":"3fd5c9c8-6304-4233-a4df-8c4c4bda5f79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":4235334318}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":4235334318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6816","nonce":4235334318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":4235334318}]},"public_addr":"172.21.15.94:6813/4235334318","cluster_addr":"172.21.15.94:6814/4235334318","heartbeat_back_addr":"172.21.15.94:6816/4235334318","heartbeat_front_addr":"172.21.15.94:6815/4235334318","state":["exists","up"]},{"osd":4,"uuid":"93461da9-bc47-489d-835e-89fcf88c9287","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6800","nonce":2666240674}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6801","nonce":2666240674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6803","nonce":2666240674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6802","nonce":2666240674}]},"public_addr":"172.21.15.151:6800/2666240674","cluster_addr":"172.21.15.151:6801/2666240674","heartbeat_back_addr":"172.21.15.151:6803/2666240674","heartbeat_front_addr":"172.21.15.151:6802/2666240674","state":["exists","up"]},{"osd":5,"uuid":"4c72b1d7-28f6-4ee3-bb80-57ff924abc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6804","nonce":2718612978}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6805","nonce":2718612978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6807","nonce":2718612978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6806","nonce":2718612978}]},"public_addr":"172.21.15.151:6804/2718612978","cluster_addr":"172.21.15.151:6805/2718612978","heartbeat_back_addr":"172.21.15.151:6807/2718612978","heartbeat_front_addr":"172.21.15.151:6806/2718612978","state":["exists","up"]},{"osd":6,"uuid":"d852205d-6b9d-4a4f-83b4-a0e59aa37657","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6808","nonce":1497503104}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6809","nonce":1497503104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6811","nonce":1497503104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6810","nonce":1497503104}]},"public_addr":"172.21.15.151:6808/1497503104","cluster_addr":"172.21.15.151:6809/1497503104","heartbeat_back_addr":"172.21.15.151:6811/1497503104","heartbeat_front_addr":"172.21.15.151:6810/1497503104","state":["exists","up"]},{"osd":7,"uuid":"cd9fca37-a6f2-46fc-90de-cd3790ad3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6812","nonce":683127336}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6813","nonce":683127336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6815","nonce":683127336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.151:6814","nonce":683127336}]},"public_addr":"172.21.15.151:6812/683127336","cluster_addr":"172.21.15.151:6813/683127336","heartbeat_back_addr":"172.21.15.151:6815/683127336","heartbeat_front_addr":"172.21.15.151:6814/683127336","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:10:26.284321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:10:52.628310+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:11:18.278275+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:11:44.195150+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:12:07.431888+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:12:32.407393+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:12:57.133251+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:13:23.236080+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/1656942945":"2024-03-13T03:14:16.504434+0000","172.21.15.94:0/1893861386":"2024-03-13T03:14:16.504434+0000","172.21.15.94:6800/1299282424":"2024-03-13T03:14:16.504434+0000","172.21.15.94:0/2533680730":"2024-03-13T03:14:16.504434+0000","172.21.15.94:0/2796836265":"2024-03-13T03:09:02.315045+0000","172.21.15.94:0/1546784249":"2024-03-13T03:14:16.504434+0000","172.21.15.94:0/1664309417":"2024-03-13T03:08:34.850802+0000","172.21.15.94:0/1936764519":"2024-03-13T03:14:16.504434+0000","172.21.15.94:0/3802544616":"2024-03-13T03:08:34.850802+0000","172.21.15.94:6800/3548280253":"2024-03-13T03:09:02.315045+0000","172.21.15.94:6800/2955156665":"2024-03-13T03:08:34.850802+0000","172.21.15.94:0/782403058":"2024-03-13T03:08:34.850802+0000","172.21.15.94:0/3132181":"2024-03-13T03:09:02.315045+0000","172.21.15.94:0/3815053600":"2024-03-13T03:09:02.315045+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:14:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.476+0000 7f7e2e848700 1 -- 172.21.15.94:0/3307199808 >> v1:172.21.15.151:6789/0 conn(0x7f7e28102d80 legacy=0x7f7e2810a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.476+0000 7f7e2e848700 1 -- 172.21.15.94:0/3307199808 shutdown_connections 2024-03-12T03:14:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.476+0000 7f7e2e848700 1 -- 172.21.15.94:0/3307199808 >> 172.21.15.94:0/3307199808 conn(0x7f7e280fc5d0 msgr2=0x7f7e281022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.476+0000 7f7e2e848700 1 -- 172.21.15.94:0/3307199808 shutdown_connections 2024-03-12T03:14:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:17.476+0000 7f7e2e848700 1 -- 172.21.15.94:0/3307199808 wait complete. 2024-03-12T03:14:17.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-12T03:14:17.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: Active manager daemon y restarted 2024-03-12T03:14:17.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: Activating manager daemon y 2024-03-12T03:14:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: osdmap e72: 8 total, 8 up, 8 in 2024-03-12T03:14:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: mgrmap e19: y(active, starting, since 0.00762422s), standbys: x 2024-03-12T03:14:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:14:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:14:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:14:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:14:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:14:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:14:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:14:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:14:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:14:17.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:14:17.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:14:17.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:14:17.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:14:17.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:14:17.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:14:17.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:14:17.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: Manager daemon y is now available 2024-03-12T03:14:17.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:14:17.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:14:17.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:17.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:17.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:14:17.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:14:17.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:14:17.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:14:17.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:14:17.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-12T03:14:17.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: Active manager daemon y restarted 2024-03-12T03:14:17.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: Activating manager daemon y 2024-03-12T03:14:17.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: osdmap e72: 8 total, 8 up, 8 in 2024-03-12T03:14:17.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: mgrmap e19: y(active, starting, since 0.00762422s), standbys: x 2024-03-12T03:14:17.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:14:17.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:14:17.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:14:17.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:14:17.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:14:17.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:14:17.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:14:17.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:14:17.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:14:17.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:14:17.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:14:17.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:14:17.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:14:17.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:14:17.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:14:17.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:14:17.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: Manager daemon y is now available 2024-03-12T03:14:17.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:14:17.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:14:17.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:17.717 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:17.717 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:14:17.717 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:14:17.717 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:14:17.717 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:14:17.717 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:14:17.718 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:14:17.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-12T03:14:17.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: Active manager daemon y restarted 2024-03-12T03:14:17.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: Activating manager daemon y 2024-03-12T03:14:17.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: osdmap e72: 8 total, 8 up, 8 in 2024-03-12T03:14:17.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: mgrmap e19: y(active, starting, since 0.00762422s), standbys: x 2024-03-12T03:14:17.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:14:17.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:14:17.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:14:17.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:14:17.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:14:17.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:14:17.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:14:17.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:14:17.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:14:17.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:14:17.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:14:17.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:14:17.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:14:17.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:14:17.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:14:17.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:14:17.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: Manager daemon y is now available 2024-03-12T03:14:17.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:14:17.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:14:17.721 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:17.721 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:17.721 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:14:17.721 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:14:17.721 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:14:17.721 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:14:17.721 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:14:18.169 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-12T03:14:18.169 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-12T03:14:18.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-12T03:14:18.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-12T03:14:18.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-12T03:14:18.171 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-12T03:14:18.171 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-12T03:14:18.171 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-12T03:14:18.401 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[29291]: [12/Mar/2024:03:14:17] ENGINE Bus STARTING 2024-03-12T03:14:18.401 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[29291]: [12/Mar/2024:03:14:17] ENGINE Serving on https://172.21.15.94:7150 2024-03-12T03:14:18.401 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3307199808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:14:18.401 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[29291]: [12/Mar/2024:03:14:17] ENGINE Serving on http://172.21.15.94:8765 2024-03-12T03:14:18.401 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[29291]: [12/Mar/2024:03:14:17] ENGINE Bus STARTED 2024-03-12T03:14:18.401 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[29291]: mgrmap e20: y(active, since 1.01514s), standbys: x 2024-03-12T03:14:18.401 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:18.401 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[29291]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:14:18.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[40484]: [12/Mar/2024:03:14:17] ENGINE Bus STARTING 2024-03-12T03:14:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[40484]: [12/Mar/2024:03:14:17] ENGINE Serving on https://172.21.15.94:7150 2024-03-12T03:14:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3307199808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:14:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[40484]: [12/Mar/2024:03:14:17] ENGINE Serving on http://172.21.15.94:8765 2024-03-12T03:14:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[40484]: [12/Mar/2024:03:14:17] ENGINE Bus STARTED 2024-03-12T03:14:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[40484]: mgrmap e20: y(active, since 1.01514s), standbys: x 2024-03-12T03:14:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:18 smithi094 ceph-mon[40484]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:14:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:18 smithi151 ceph-mon[32923]: [12/Mar/2024:03:14:17] ENGINE Bus STARTING 2024-03-12T03:14:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:18 smithi151 ceph-mon[32923]: [12/Mar/2024:03:14:17] ENGINE Serving on https://172.21.15.94:7150 2024-03-12T03:14:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3307199808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:14:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:18 smithi151 ceph-mon[32923]: [12/Mar/2024:03:14:17] ENGINE Serving on http://172.21.15.94:8765 2024-03-12T03:14:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:18 smithi151 ceph-mon[32923]: [12/Mar/2024:03:14:17] ENGINE Bus STARTED 2024-03-12T03:14:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:18 smithi151 ceph-mon[32923]: mgrmap e20: y(active, since 1.01514s), standbys: x 2024-03-12T03:14:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:18 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:18 smithi151 ceph-mon[32923]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:14:19.032 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:19.033 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:19.034 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:19.039 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:19.442 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:19 smithi094 ceph-mon[29291]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/690 objects degraded (1.014%), 1 pg degraded) 2024-03-12T03:14:19.442 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:19 smithi094 ceph-mon[29291]: Cluster is now healthy 2024-03-12T03:14:19.442 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:19 smithi094 ceph-mon[29291]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:14:19.442 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:19 smithi094 ceph-mon[40484]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/690 objects degraded (1.014%), 1 pg degraded) 2024-03-12T03:14:19.442 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:19 smithi094 ceph-mon[40484]: Cluster is now healthy 2024-03-12T03:14:19.442 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:19 smithi094 ceph-mon[40484]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:14:19.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:19 smithi151 ceph-mon[32923]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/690 objects degraded (1.014%), 1 pg degraded) 2024-03-12T03:14:19.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:19 smithi151 ceph-mon[32923]: Cluster is now healthy 2024-03-12T03:14:19.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:19 smithi151 ceph-mon[32923]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:14:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:20 smithi094 ceph-mon[29291]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-12T03:14:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:20 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:20 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:20.687 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:20 smithi094 ceph-mon[40484]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-12T03:14:20.688 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:20 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:20.688 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:20 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:20.688 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:14:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:14:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:20 smithi151 ceph-mon[32923]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-12T03:14:20.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:20 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:20.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:20 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:21.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[29291]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:14:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:21.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[40484]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:14:21.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:21.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:21.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:21 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:21 smithi151 ceph-mon[32923]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:14:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:21 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:21 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:21 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:21 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:21 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:21 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:21 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:21 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:21 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:21 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:21.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:21 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:22.833 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:22 smithi094 ceph-mon[29291]: Adjusting osd_memory_target on smithi151 to 3760M 2024-03-12T03:14:22.833 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:22 smithi094 ceph-mon[29291]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-12T03:14:22.833 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:22 smithi094 ceph-mon[29291]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:14:22.834 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:22 smithi094 ceph-mon[40484]: Adjusting osd_memory_target on smithi151 to 3760M 2024-03-12T03:14:22.834 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:22 smithi094 ceph-mon[40484]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-12T03:14:22.834 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:22 smithi094 ceph-mon[40484]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:14:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:22 smithi151 ceph-mon[32923]: Adjusting osd_memory_target on smithi151 to 3760M 2024-03-12T03:14:22.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:22 smithi151 ceph-mon[32923]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-12T03:14:22.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:22 smithi151 ceph-mon[32923]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:14:24.295 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.296+0000 7fd4f4b43700 1 -- 172.21.15.94:0/4285820032 >> v1:172.21.15.94:6789/0 conn(0x7fd4f0106a70 legacy=0x7fd4f0106e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.296+0000 7fd4f4b43700 1 -- 172.21.15.94:0/4285820032 shutdown_connections 2024-03-12T03:14:24.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.296+0000 7fd4f4b43700 1 -- 172.21.15.94:0/4285820032 >> 172.21.15.94:0/4285820032 conn(0x7fd4f00fc580 msgr2=0x7fd4f00fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:24.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.296+0000 7fd4f4b43700 1 -- 172.21.15.94:0/4285820032 shutdown_connections 2024-03-12T03:14:24.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.296+0000 7fd4f4b43700 1 -- 172.21.15.94:0/4285820032 wait complete. 2024-03-12T03:14:24.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4f4b43700 1 Processor -- start 2024-03-12T03:14:24.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4f4b43700 1 -- start start 2024-03-12T03:14:24.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4f4b43700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd4f00fe530 con 0x7fd4f0106a70 2024-03-12T03:14:24.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4f4b43700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd4f00fe740 con 0x7fd4f010d4f0 2024-03-12T03:14:24.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4f4b43700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd4f00fe970 con 0x7fd4f0109920 2024-03-12T03:14:24.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4ee59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd4f0106a70 0x7fd4f00fca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43330/0 (socket says 172.21.15.94:43330) 2024-03-12T03:14:24.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4edd9b700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fd4f0109920 0x7fd4f00fd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:57406/0 (socket says 172.21.15.94:57406) 2024-03-12T03:14:24.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4ee59c700 1 -- 172.21.15.94:0/176144674 learned_addr learned my addr 172.21.15.94:0/176144674 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:24.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175686790 0 0) 0x7fd4f00fe530 con 0x7fd4f0106a70 2024-03-12T03:14:24.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d80035b0 con 0x7fd4f0106a70 2024-03-12T03:14:24.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818738332 0 0) 0x7fd4f00fe970 con 0x7fd4f0109920 2024-03-12T03:14:24.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f00fe530 con 0x7fd4f0109920 2024-03-12T03:14:24.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729460529 0 0) 0x7fd4f00fe740 con 0x7fd4f010d4f0 2024-03-12T03:14:24.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.297+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f00fe970 con 0x7fd4f010d4f0 2024-03-12T03:14:24.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3801758449 0 0) 0x7fd4d80035b0 con 0x7fd4f0106a70 2024-03-12T03:14:24.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4f00fe740 con 0x7fd4f0106a70 2024-03-12T03:14:24.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fd4e00044a0 con 0x7fd4f0106a70 2024-03-12T03:14:24.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 224576674 0 0) 0x7fd4f00fe530 con 0x7fd4f0109920 2024-03-12T03:14:24.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4d80035b0 con 0x7fd4f0109920 2024-03-12T03:14:24.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fd4e4001830 con 0x7fd4f0109920 2024-03-12T03:14:24.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1744034524 0 0) 0x7fd4f00fe970 con 0x7fd4f010d4f0 2024-03-12T03:14:24.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4f00fe530 con 0x7fd4f010d4f0 2024-03-12T03:14:24.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fd4e8001470 con 0x7fd4f010d4f0 2024-03-12T03:14:24.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 795866319 0 0) 0x7fd4f00fe740 con 0x7fd4f0106a70 2024-03-12T03:14:24.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 >> v1:172.21.15.94:6790/0 conn(0x7fd4f0109920 legacy=0x7fd4f00fd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 >> v1:172.21.15.151:6789/0 conn(0x7fd4f010d4f0 legacy=0x7fd4f01a7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f01ac560 con 0x7fd4f0106a70 2024-03-12T03:14:24.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4f4b43700 1 -- 172.21.15.94:0/176144674 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f01a9550 con 0x7fd4f0106a70 2024-03-12T03:14:24.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4f4b43700 1 -- 172.21.15.94:0/176144674 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f01a9a10 con 0x7fd4f0106a70 2024-03-12T03:14:24.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fd4e0004700 con 0x7fd4f0106a70 2024-03-12T03:14:24.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.298+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fd4e0004c40 con 0x7fd4f0106a70 2024-03-12T03:14:24.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.299+0000 7fd4f4b43700 1 -- 172.21.15.94:0/176144674 --> v1:172.21.15.94:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd4d0000f80 con 0x7fd4f0106a70 2024-03-12T03:14:24.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.300+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7fd4e0004ec0 con 0x7fd4f0106a70 2024-03-12T03:14:24.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.300+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7fd4e0089320 con 0x7fd4f0106a70 2024-03-12T03:14:24.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.300+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 --> v1:172.21.15.94:6813/4235334318 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4e0001080 con 0x7fd4d8079570 2024-03-12T03:14:24.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.300+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7fd4e0089730 con 0x7fd4f0106a70 2024-03-12T03:14:24.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.301+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== osd.3 v1:172.21.15.94:6813/4235334318 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd4e0001080 con 0x7fd4d8079570 2024-03-12T03:14:24.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.319+0000 7fd4f4b43700 1 -- 172.21.15.94:0/176144674 --> v1:172.21.15.94:6813/4235334318 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd4d0002ce0 con 0x7fd4d8079570 2024-03-12T03:14:24.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.320+0000 7fd4df7fe700 1 -- 172.21.15.94:0/176144674 <== osd.3 v1:172.21.15.94:6813/4235334318 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 902111241) 0x7fd4d0002ce0 con 0x7fd4d8079570 2024-03-12T03:14:24.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.320+0000 7fd4f4b43700 1 -- 172.21.15.94:0/176144674 >> v1:172.21.15.94:6813/4235334318 conn(0x7fd4d8079570 legacy=0x7fd4d807ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.320+0000 7fd4f4b43700 1 -- 172.21.15.94:0/176144674 >> v1:172.21.15.94:6800/2099808905 conn(0x7fd4d806ed10 legacy=0x7fd4d80711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.320+0000 7fd4f4b43700 1 -- 172.21.15.94:0/176144674 >> v1:172.21.15.94:6789/0 conn(0x7fd4f0106a70 legacy=0x7fd4f00fca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.320+0000 7fd4eed9d700 1 -- 172.21.15.94:0/176144674 reap_dead start 2024-03-12T03:14:24.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.320+0000 7fd4f4b43700 1 -- 172.21.15.94:0/176144674 shutdown_connections 2024-03-12T03:14:24.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.320+0000 7fd4f4b43700 1 -- 172.21.15.94:0/176144674 >> 172.21.15.94:0/176144674 conn(0x7fd4f00fc580 msgr2=0x7fd4f0100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:24.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.320+0000 7fd4f4b43700 1 -- 172.21.15.94:0/176144674 shutdown_connections 2024-03-12T03:14:24.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.320+0000 7fd4f4b43700 1 -- 172.21.15.94:0/176144674 wait complete. 2024-03-12T03:14:24.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.496+0000 7fea98392700 1 -- 172.21.15.94:0/253759772 >> v1:172.21.15.94:6789/0 conn(0x7fea90101de0 legacy=0x7fea901021c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.496+0000 7fea98392700 1 -- 172.21.15.94:0/253759772 shutdown_connections 2024-03-12T03:14:24.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.496+0000 7fea98392700 1 -- 172.21.15.94:0/253759772 >> 172.21.15.94:0/253759772 conn(0x7fea900fd690 msgr2=0x7fea900ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:24.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.496+0000 7fea98392700 1 -- 172.21.15.94:0/253759772 shutdown_connections 2024-03-12T03:14:24.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.496+0000 7fea98392700 1 -- 172.21.15.94:0/253759772 wait complete. 2024-03-12T03:14:24.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.497+0000 7fea98392700 1 Processor -- start 2024-03-12T03:14:24.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.497+0000 7fea98392700 1 -- start start 2024-03-12T03:14:24.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.497+0000 7fea98392700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fea901a6c10 con 0x7fea90101de0 2024-03-12T03:14:24.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.497+0000 7fea98392700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fea901a7e30 con 0x7fea9010aa70 2024-03-12T03:14:24.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.497+0000 7fea98392700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fea901a9070 con 0x7fea9010e610 2024-03-12T03:14:24.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.497+0000 7fea9692f700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fea9010e610 0x7fea901a52d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:56914/0 (socket says 172.21.15.94:56914) 2024-03-12T03:14:24.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.497+0000 7fea9612e700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fea90101de0 0x7fea901070b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:60822/0 (socket says 172.21.15.94:60822) 2024-03-12T03:14:24.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.497+0000 7fea9692f700 1 -- 172.21.15.94:0/1651002289 learned_addr learned my addr 172.21.15.94:0/1651002289 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:24.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332674717 0 0) 0x7fea901a9070 con 0x7fea9010e610 2024-03-12T03:14:24.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7c0035b0 con 0x7fea9010e610 2024-03-12T03:14:24.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005010256 0 0) 0x7fea901a6c10 con 0x7fea90101de0 2024-03-12T03:14:24.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea901a9070 con 0x7fea90101de0 2024-03-12T03:14:24.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834371752 0 0) 0x7fea901a7e30 con 0x7fea9010aa70 2024-03-12T03:14:24.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea901a6c10 con 0x7fea9010aa70 2024-03-12T03:14:24.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1673551603 0 0) 0x7fea7c0035b0 con 0x7fea9010e610 2024-03-12T03:14:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea901a7e30 con 0x7fea9010e610 2024-03-12T03:14:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fea88001c60 con 0x7fea9010e610 2024-03-12T03:14:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1742902116 0 0) 0x7fea901a9070 con 0x7fea90101de0 2024-03-12T03:14:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea7c0035b0 con 0x7fea90101de0 2024-03-12T03:14:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fea8c001510 con 0x7fea90101de0 2024-03-12T03:14:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 929577855 0 0) 0x7fea901a6c10 con 0x7fea9010aa70 2024-03-12T03:14:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea901a9070 con 0x7fea9010aa70 2024-03-12T03:14:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fea800016a0 con 0x7fea9010aa70 2024-03-12T03:14:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 636791565 0 0) 0x7fea901a7e30 con 0x7fea9010e610 2024-03-12T03:14:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 >> v1:172.21.15.151:6789/0 conn(0x7fea9010aa70 legacy=0x7fea901a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 >> v1:172.21.15.94:6789/0 conn(0x7fea90101de0 legacy=0x7fea901070b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea901aa2b0 con 0x7fea9010e610 2024-03-12T03:14:24.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea98392700 1 -- 172.21.15.94:0/1651002289 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea901a92a0 con 0x7fea9010e610 2024-03-12T03:14:24.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea98392700 1 -- 172.21.15.94:0/1651002289 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea901a9760 con 0x7fea9010e610 2024-03-12T03:14:24.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fea88002000 con 0x7fea9010e610 2024-03-12T03:14:24.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.498+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fea88006050 con 0x7fea9010e610 2024-03-12T03:14:24.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.499+0000 7fea98392700 1 -- 172.21.15.94:0/1651002289 --> v1:172.21.15.94:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fea74000f80 con 0x7fea9010e610 2024-03-12T03:14:24.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.500+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7fea88003cb0 con 0x7fea9010e610 2024-03-12T03:14:24.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.500+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7fea88089400 con 0x7fea9010e610 2024-03-12T03:14:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.500+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 --> v1:172.21.15.151:6808/1497503104 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea88004620 con 0x7fea7c0795a0 2024-03-12T03:14:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.500+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7fea88089820 con 0x7fea9010e610 2024-03-12T03:14:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.502+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== osd.6 v1:172.21.15.151:6808/1497503104 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fea88004620 con 0x7fea7c0795a0 2024-03-12T03:14:24.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.517+0000 7fea98392700 1 -- 172.21.15.94:0/1651002289 --> v1:172.21.15.151:6808/1497503104 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fea74002ce0 con 0x7fea7c0795a0 2024-03-12T03:14:24.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.518+0000 7fea877fe700 1 -- 172.21.15.94:0/1651002289 <== osd.6 v1:172.21.15.151:6808/1497503104 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 2319438110) 0x7fea74002ce0 con 0x7fea7c0795a0 2024-03-12T03:14:24.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.518+0000 7fea98392700 1 -- 172.21.15.94:0/1651002289 >> v1:172.21.15.151:6808/1497503104 conn(0x7fea7c0795a0 legacy=0x7fea7c07ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.518+0000 7fea98392700 1 -- 172.21.15.94:0/1651002289 >> v1:172.21.15.94:6800/2099808905 conn(0x7fea7c06edc0 legacy=0x7fea7c071280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.518+0000 7fea98392700 1 -- 172.21.15.94:0/1651002289 >> v1:172.21.15.94:6790/0 conn(0x7fea9010e610 legacy=0x7fea901a52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.518+0000 7fea9692f700 1 -- 172.21.15.94:0/1651002289 reap_dead start 2024-03-12T03:14:24.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.519+0000 7fea98392700 1 -- 172.21.15.94:0/1651002289 shutdown_connections 2024-03-12T03:14:24.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.519+0000 7fea98392700 1 -- 172.21.15.94:0/1651002289 >> 172.21.15.94:0/1651002289 conn(0x7fea900fd690 msgr2=0x7fea90107940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:24.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.519+0000 7fea98392700 1 -- 172.21.15.94:0/1651002289 shutdown_connections 2024-03-12T03:14:24.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.519+0000 7fea98392700 1 -- 172.21.15.94:0/1651002289 wait complete. 2024-03-12T03:14:24.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.689+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/1118833322 >> v1:172.21.15.94:6790/0 conn(0x7f1dcc10aa60 legacy=0x7f1dcc10ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.689+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/1118833322 shutdown_connections 2024-03-12T03:14:24.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.689+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/1118833322 >> 172.21.15.94:0/1118833322 conn(0x7f1dcc076f10 msgr2=0x7f1dcc0fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:24.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.689+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/1118833322 shutdown_connections 2024-03-12T03:14:24.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.689+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/1118833322 wait complete. 2024-03-12T03:14:24.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.690+0000 7f1dd0aa7700 1 Processor -- start 2024-03-12T03:14:24.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.690+0000 7f1dd0aa7700 1 -- start start 2024-03-12T03:14:24.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.690+0000 7f1dd0aa7700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1dcc1a6c00 con 0x7f1dcc10aa60 2024-03-12T03:14:24.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.690+0000 7f1dd0aa7700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1dcc1a7e20 con 0x7f1dcc1002e0 2024-03-12T03:14:24.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.690+0000 7f1dd0aa7700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1dcc1a9060 con 0x7f1dcc10e600 2024-03-12T03:14:24.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.690+0000 7f1dc9d9b700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1dcc10aa60 0x7f1dcc1a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:60840/0 (socket says 172.21.15.94:60840) 2024-03-12T03:14:24.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.690+0000 7f1dcad9d700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f1dcc10e600 0x7f1dcc1a52c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:56918/0 (socket says 172.21.15.94:56918) 2024-03-12T03:14:24.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.690+0000 7f1dc9d9b700 1 -- 172.21.15.94:0/373393395 learned_addr learned my addr 172.21.15.94:0/373393395 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:24.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688675954 0 0) 0x7f1dcc1a9060 con 0x7f1dcc10e600 2024-03-12T03:14:24.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db40035b0 con 0x7f1dcc10e600 2024-03-12T03:14:24.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202392328 0 0) 0x7f1dcc1a6c00 con 0x7f1dcc10aa60 2024-03-12T03:14:24.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dcc1a9060 con 0x7f1dcc10aa60 2024-03-12T03:14:24.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801179714 0 0) 0x7f1dcc1a7e20 con 0x7f1dcc1002e0 2024-03-12T03:14:24.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dcc1a6c00 con 0x7f1dcc1002e0 2024-03-12T03:14:24.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1515067607 0 0) 0x7f1dcc1a9060 con 0x7f1dcc10aa60 2024-03-12T03:14:24.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dcc1a7e20 con 0x7f1dcc10aa60 2024-03-12T03:14:24.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1dbc004370 con 0x7f1dcc10aa60 2024-03-12T03:14:24.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2270184218 0 0) 0x7f1db40035b0 con 0x7f1dcc10e600 2024-03-12T03:14:24.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dcc1a9060 con 0x7f1dcc10e600 2024-03-12T03:14:24.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3360110520 0 0) 0x7f1dcc1a6c00 con 0x7f1dcc1002e0 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1db40035b0 con 0x7f1dcc1002e0 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1dc4001c10 con 0x7f1dcc10e600 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1dc0001830 con 0x7f1dcc1002e0 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3461801298 0 0) 0x7f1dcc1a7e20 con 0x7f1dcc10aa60 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 >> v1:172.21.15.94:6790/0 conn(0x7f1dcc10e600 legacy=0x7f1dcc1a52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 >> v1:172.21.15.151:6789/0 conn(0x7f1dcc1002e0 legacy=0x7f1dcc1070a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc1aa2a0 con 0x7f1dcc10aa60 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/373393395 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dcc1a9290 con 0x7f1dcc10aa60 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/373393395 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dcc1a9750 con 0x7f1dcc10aa60 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f1dbc001ab0 con 0x7f1dcc10aa60 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.691+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1dbc004cb0 con 0x7f1dcc10aa60 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.692+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/373393395 --> v1:172.21.15.94:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1dac000f80 con 0x7f1dcc10aa60 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.693+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f1dbc003710 con 0x7f1dcc10aa60 2024-03-12T03:14:24.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.693+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7f1dbc089140 con 0x7f1dcc10aa60 2024-03-12T03:14:24.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.693+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 --> v1:172.21.15.151:6812/683127336 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1dbc0057b0 con 0x7f1db4079640 2024-03-12T03:14:24.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.693+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7f1dbc089550 con 0x7f1dcc10aa60 2024-03-12T03:14:24.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.695+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== osd.7 v1:172.21.15.151:6812/683127336 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1dbc0057b0 con 0x7f1db4079640 2024-03-12T03:14:24.707 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:24.713 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:24 smithi151 ceph-mon[32923]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:14:24.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.713+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/373393395 --> v1:172.21.15.151:6812/683127336 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1dac002ce0 con 0x7f1db4079640 2024-03-12T03:14:24.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:24 smithi094 ceph-mon[29291]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:14:24.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:24 smithi094 ceph-mon[40484]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:14:24.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.714+0000 7f1dbb7fe700 1 -- 172.21.15.94:0/373393395 <== osd.7 v1:172.21.15.151:6812/683127336 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 1422466053) 0x7f1dac002ce0 con 0x7f1db4079640 2024-03-12T03:14:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.715+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/373393395 >> v1:172.21.15.151:6812/683127336 conn(0x7f1db4079640 legacy=0x7f1db407bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.715+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/373393395 >> v1:172.21.15.94:6800/2099808905 conn(0x7f1db406edd0 legacy=0x7f1db4071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.715+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/373393395 >> v1:172.21.15.94:6789/0 conn(0x7f1dcc10aa60 legacy=0x7f1dcc1a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.715+0000 7f1dcad9d700 1 -- 172.21.15.94:0/373393395 reap_dead start 2024-03-12T03:14:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.715+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/373393395 shutdown_connections 2024-03-12T03:14:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.715+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/373393395 >> 172.21.15.94:0/373393395 conn(0x7f1dcc076f10 msgr2=0x7f1dcc111270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.715+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/373393395 shutdown_connections 2024-03-12T03:14:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:24.716+0000 7f1dd0aa7700 1 -- 172.21.15.94:0/373393395 wait complete. 2024-03-12T03:14:25.446 INFO:teuthology.orchestra.run.smithi094.stdout:197568495634 2024-03-12T03:14:25.446 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-12T03:14:26.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.285+0000 7fc66e733700 1 -- 172.21.15.94:0/4138019842 >> v1:172.21.15.94:6789/0 conn(0x7fc66810e6b0 legacy=0x7fc668110b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:26.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.286+0000 7fc66e733700 1 -- 172.21.15.94:0/4138019842 shutdown_connections 2024-03-12T03:14:26.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.286+0000 7fc66e733700 1 -- 172.21.15.94:0/4138019842 >> 172.21.15.94:0/4138019842 conn(0x7fc6680fc580 msgr2=0x7fc6680fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:26.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.286+0000 7fc66e733700 1 -- 172.21.15.94:0/4138019842 shutdown_connections 2024-03-12T03:14:26.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.286+0000 7fc66e733700 1 -- 172.21.15.94:0/4138019842 wait complete. 2024-03-12T03:14:26.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.286+0000 7fc66e733700 1 Processor -- start 2024-03-12T03:14:26.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.286+0000 7fc66e733700 1 -- start start 2024-03-12T03:14:26.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.287+0000 7fc66e733700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6681a6a40 con 0x7fc66810aba0 2024-03-12T03:14:26.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.287+0000 7fc66e733700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6681a7c60 con 0x7fc66810e6b0 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.287+0000 7fc6677fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc66810aba0 0x7fc6681a1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:60854/0 (socket says 172.21.15.94:60854) 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.287+0000 7fc6677fe700 1 -- 172.21.15.94:0/1352422198 learned_addr learned my addr 172.21.15.94:0/1352422198 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.287+0000 7fc66e733700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6681a8e80 con 0x7fc668102d90 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.287+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303280869 0 0) 0x7fc6681a6a40 con 0x7fc66810aba0 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.287+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6500031f0 con 0x7fc66810aba0 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.287+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636556699 0 0) 0x7fc6681a8e80 con 0x7fc668102d90 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.287+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6681a6a40 con 0x7fc668102d90 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476696606 0 0) 0x7fc6681a7c60 con 0x7fc66810e6b0 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6681a8e80 con 0x7fc66810e6b0 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4052199273 0 0) 0x7fc6500031f0 con 0x7fc66810aba0 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6681a7c60 con 0x7fc66810aba0 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc65c0019c0 con 0x7fc66810aba0 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2035765446 0 0) 0x7fc6681a6a40 con 0x7fc668102d90 2024-03-12T03:14:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6500031f0 con 0x7fc668102d90 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc658001960 con 0x7fc668102d90 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3726380127 0 0) 0x7fc6681a7c60 con 0x7fc66810aba0 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 >> v1:172.21.15.94:6790/0 conn(0x7fc668102d90 legacy=0x7fc66810a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 >> v1:172.21.15.151:6789/0 conn(0x7fc66810e6b0 legacy=0x7fc6681a51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6681aa0c0 con 0x7fc66810aba0 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc66e733700 1 -- 172.21.15.94:0/1352422198 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6681a90b0 con 0x7fc66810aba0 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc66e733700 1 -- 172.21.15.94:0/1352422198 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6681a9570 con 0x7fc66810aba0 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fc65c001ca0 con 0x7fc66810aba0 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.288+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc65c005dd0 con 0x7fc66810aba0 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.289+0000 7fc66e733700 1 -- 172.21.15.94:0/1352422198 --> v1:172.21.15.94:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc654000f80 con 0x7fc66810aba0 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.290+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7fc65c01c460 con 0x7fc66810aba0 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.290+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7fc65c089170 con 0x7fc66810aba0 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.291+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 --> v1:172.21.15.94:6801/2935792501 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc65c01cdd0 con 0x7fc650079110 2024-03-12T03:14:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.291+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7fc65c0895b0 con 0x7fc66810aba0 2024-03-12T03:14:26.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.292+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== osd.0 v1:172.21.15.94:6801/2935792501 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc65c01cdd0 con 0x7fc650079110 2024-03-12T03:14:26.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.309+0000 7fc66e733700 1 -- 172.21.15.94:0/1352422198 --> v1:172.21.15.94:6801/2935792501 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc654002ce0 con 0x7fc650079110 2024-03-12T03:14:26.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.309+0000 7fc6657fa700 1 -- 172.21.15.94:0/1352422198 <== osd.0 v1:172.21.15.94:6801/2935792501 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1002017476) 0x7fc654002ce0 con 0x7fc650079110 2024-03-12T03:14:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.310+0000 7fc66e733700 1 -- 172.21.15.94:0/1352422198 >> v1:172.21.15.94:6801/2935792501 conn(0x7fc650079110 legacy=0x7fc65007b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.310+0000 7fc66e733700 1 -- 172.21.15.94:0/1352422198 >> v1:172.21.15.94:6800/2099808905 conn(0x7fc65006ea30 legacy=0x7fc650070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.310+0000 7fc66e733700 1 -- 172.21.15.94:0/1352422198 >> v1:172.21.15.94:6789/0 conn(0x7fc66810aba0 legacy=0x7fc6681a1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.310+0000 7fc66ccd0700 1 -- 172.21.15.94:0/1352422198 reap_dead start 2024-03-12T03:14:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.310+0000 7fc66e733700 1 -- 172.21.15.94:0/1352422198 shutdown_connections 2024-03-12T03:14:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.310+0000 7fc66e733700 1 -- 172.21.15.94:0/1352422198 >> 172.21.15.94:0/1352422198 conn(0x7fc6680fc580 msgr2=0x7fc6680fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.310+0000 7fc66e733700 1 -- 172.21.15.94:0/1352422198 shutdown_connections 2024-03-12T03:14:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:26.310+0000 7fc66e733700 1 -- 172.21.15.94:0/1352422198 wait complete. 2024-03-12T03:14:27.149 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:26 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:27.149 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:26 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:27.150 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:26 smithi094 ceph-mon[29291]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:14:27.150 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:26 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:27.150 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:26 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:27.150 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:26 smithi094 ceph-mon[40484]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:14:27.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:26 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:27.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:26 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:27.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:26 smithi151 ceph-mon[32923]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:14:27.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:14:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:14:28.185 INFO:teuthology.orchestra.run.smithi094.stdout:231928233997 2024-03-12T03:14:28.186 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-12T03:14:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:28 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:28 smithi094 ceph-mon[29291]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:14:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:28 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:28 smithi094 ceph-mon[40484]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:14:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:28 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:28 smithi151 ceph-mon[32923]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:14:29.367 INFO:teuthology.orchestra.run.smithi094.stdout:107374182432 2024-03-12T03:14:29.367 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-12T03:14:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: Adjusting osd_memory_target on smithi094 to 3760M 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: Updating smithi094:/etc/ceph/ceph.conf 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: Updating smithi094:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: Updating smithi094:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:14:29.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:14:29.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: Adjusting osd_memory_target on smithi094 to 3760M 2024-03-12T03:14:29.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:14:29.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:14:29.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: Updating smithi094:/etc/ceph/ceph.conf 2024-03-12T03:14:29.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-12T03:14:29.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: Updating smithi094:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:14:29.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:14:29.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:14:29.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:14:29.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: Updating smithi094:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:14:29.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:14:29.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: Adjusting osd_memory_target on smithi094 to 3760M 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: Updating smithi094:/etc/ceph/ceph.conf 2024-03-12T03:14:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-12T03:14:29.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: Updating smithi094:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:14:29.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:14:29.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:14:29.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:14:29.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: Updating smithi094:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:14:29.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: Updating smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:14:29.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:29.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:30.326 INFO:teuthology.orchestra.run.smithi094.stdout:34359738418 2024-03-12T03:14:30.327 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-12T03:14:30.609 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:14:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:14:30.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.610+0000 7f97ff8d3700 1 -- 172.21.15.94:0/1231344716 >> v1:172.21.15.94:6789/0 conn(0x7f97f8100450 legacy=0x7f97f8100830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:30.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.610+0000 7f97ff8d3700 1 -- 172.21.15.94:0/1231344716 shutdown_connections 2024-03-12T03:14:30.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.610+0000 7f97ff8d3700 1 -- 172.21.15.94:0/1231344716 >> 172.21.15.94:0/1231344716 conn(0x7f97f8076f10 msgr2=0x7f97f80fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:30.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.610+0000 7f97ff8d3700 1 -- 172.21.15.94:0/1231344716 shutdown_connections 2024-03-12T03:14:30.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.610+0000 7f97ff8d3700 1 -- 172.21.15.94:0/1231344716 wait complete. 2024-03-12T03:14:30.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.611+0000 7f97ff8d3700 1 Processor -- start 2024-03-12T03:14:30.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.611+0000 7f97ff8d3700 1 -- start start 2024-03-12T03:14:30.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.611+0000 7f97ff8d3700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f97f81a6c40 con 0x7f97f810cc80 2024-03-12T03:14:30.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.611+0000 7f97ff8d3700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f97f81a7e60 con 0x7f97f8100450 2024-03-12T03:14:30.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.611+0000 7f97ff8d3700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f97f81a90a0 con 0x7f97f81090e0 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.611+0000 7f97fde70700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f97f810cc80 0x7f97f81a5300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:60880/0 (socket says 172.21.15.94:60880) 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.611+0000 7f97fde70700 1 -- 172.21.15.94:0/665049317 learned_addr learned my addr 172.21.15.94:0/665049317 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.611+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288330327 0 0) 0x7f97f81a90a0 con 0x7f97f81090e0 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.611+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97e40031f0 con 0x7f97f81090e0 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.611+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455780479 0 0) 0x7f97f81a7e60 con 0x7f97f8100450 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.611+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f81a90a0 con 0x7f97f8100450 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.611+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765398348 0 0) 0x7f97f81a6c40 con 0x7f97f810cc80 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.611+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f81a7e60 con 0x7f97f810cc80 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1154610503 0 0) 0x7f97f81a90a0 con 0x7f97f8100450 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97f81a6c40 con 0x7f97f8100450 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f97f4004540 con 0x7f97f8100450 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 940779766 0 0) 0x7f97e40031f0 con 0x7f97f81090e0 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97f81a90a0 con 0x7f97f81090e0 2024-03-12T03:14:30.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3440955837 0 0) 0x7f97f81a7e60 con 0x7f97f810cc80 2024-03-12T03:14:30.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97e40031f0 con 0x7f97f810cc80 2024-03-12T03:14:30.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f97e8001830 con 0x7f97f81090e0 2024-03-12T03:14:30.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f97f0001c10 con 0x7f97f810cc80 2024-03-12T03:14:30.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 600253286 0 0) 0x7f97f81a6c40 con 0x7f97f8100450 2024-03-12T03:14:30.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 >> v1:172.21.15.94:6790/0 conn(0x7f97f81090e0 legacy=0x7f97f81049f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:30.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 >> v1:172.21.15.94:6789/0 conn(0x7f97f810cc80 legacy=0x7f97f81a5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:30.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f81aa2e0 con 0x7f97f8100450 2024-03-12T03:14:30.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ff8d3700 1 -- 172.21.15.94:0/665049317 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f81a8090 con 0x7f97f8100450 2024-03-12T03:14:30.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ff8d3700 1 -- 172.21.15.94:0/665049317 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f81a8550 con 0x7f97f8100450 2024-03-12T03:14:30.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f97f4001a40 con 0x7f97f8100450 2024-03-12T03:14:30.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.612+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f97f4004ce0 con 0x7f97f8100450 2024-03-12T03:14:30.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.613+0000 7f97ff8d3700 1 -- 172.21.15.94:0/665049317 --> v1:172.21.15.151:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f97dc000f80 con 0x7f97f8100450 2024-03-12T03:14:30.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.614+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f97f40037f0 con 0x7f97f8100450 2024-03-12T03:14:30.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.614+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7f97f4088110 con 0x7f97f8100450 2024-03-12T03:14:30.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.614+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 --> v1:172.21.15.94:6805/1056655004 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97e407d230 con 0x7f97e40795c0 2024-03-12T03:14:30.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.614+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7f97f4088550 con 0x7f97f8100450 2024-03-12T03:14:30.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.615+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== osd.1 v1:172.21.15.94:6805/1056655004 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f97e407d230 con 0x7f97e40795c0 2024-03-12T03:14:30.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.632+0000 7f97ff8d3700 1 -- 172.21.15.94:0/665049317 --> v1:172.21.15.94:6805/1056655004 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f97dc002ce0 con 0x7f97e40795c0 2024-03-12T03:14:30.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.632+0000 7f97ee7fc700 1 -- 172.21.15.94:0/665049317 <== osd.1 v1:172.21.15.94:6805/1056655004 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 4200525082) 0x7f97dc002ce0 con 0x7f97e40795c0 2024-03-12T03:14:30.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.632+0000 7f97ff8d3700 1 -- 172.21.15.94:0/665049317 >> v1:172.21.15.94:6805/1056655004 conn(0x7f97e40795c0 legacy=0x7f97e407ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:30.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.632+0000 7f97ff8d3700 1 -- 172.21.15.94:0/665049317 >> v1:172.21.15.94:6800/2099808905 conn(0x7f97e406edd0 legacy=0x7f97e4071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:30.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.632+0000 7f97ff8d3700 1 -- 172.21.15.94:0/665049317 >> v1:172.21.15.151:6789/0 conn(0x7f97f8100450 legacy=0x7f97f81042e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:30.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.632+0000 7f97fde70700 1 -- 172.21.15.94:0/665049317 reap_dead start 2024-03-12T03:14:30.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.633+0000 7f97ff8d3700 1 -- 172.21.15.94:0/665049317 shutdown_connections 2024-03-12T03:14:30.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.633+0000 7f97ff8d3700 1 -- 172.21.15.94:0/665049317 >> 172.21.15.94:0/665049317 conn(0x7f97f8076f10 msgr2=0x7f97f8108800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:30.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.633+0000 7f97ff8d3700 1 -- 172.21.15.94:0/665049317 shutdown_connections 2024-03-12T03:14:30.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:30.633+0000 7f97ff8d3700 1 -- 172.21.15.94:0/665049317 wait complete. 2024-03-12T03:14:30.859 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:30 smithi094 ceph-mon[40484]: Deploying daemon alertmanager.a on smithi094 2024-03-12T03:14:30.860 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:30 smithi094 ceph-mon[40484]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:14:30.860 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:30 smithi094 ceph-mon[29291]: Deploying daemon alertmanager.a on smithi094 2024-03-12T03:14:30.860 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:30 smithi094 ceph-mon[29291]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:14:30.864 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:30 smithi151 ceph-mon[32923]: Deploying daemon alertmanager.a on smithi094 2024-03-12T03:14:30.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:30 smithi151 ceph-mon[32923]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:14:32.460 INFO:teuthology.orchestra.run.smithi094.stdout:55834574892 2024-03-12T03:14:32.461 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-12T03:14:32.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:32 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:32 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:14:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:32 smithi094 ceph-mon[29291]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:14:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:32 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:32 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:14:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:32 smithi094 ceph-mon[40484]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:14:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:32 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:32 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:14:32.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:32 smithi151 ceph-mon[32923]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:14:33.170 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:33.838 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:34.658 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:34 smithi094 ceph-mon[40484]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:14:34.658 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:34 smithi094 ceph-mon[29291]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:14:34.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.757+0000 7f200ea61700 1 -- 172.21.15.94:0/1381790692 >> v1:172.21.15.94:6790/0 conn(0x7f2008106a70 legacy=0x7f2008106e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:34.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.758+0000 7f200ea61700 1 -- 172.21.15.94:0/1381790692 shutdown_connections 2024-03-12T03:14:34.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.758+0000 7f200ea61700 1 -- 172.21.15.94:0/1381790692 >> 172.21.15.94:0/1381790692 conn(0x7f20080fc580 msgr2=0x7f20080fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:34.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.758+0000 7f200ea61700 1 -- 172.21.15.94:0/1381790692 shutdown_connections 2024-03-12T03:14:34.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.758+0000 7f200ea61700 1 -- 172.21.15.94:0/1381790692 wait complete. 2024-03-12T03:14:34.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.758+0000 7f200ea61700 1 Processor -- start 2024-03-12T03:14:34.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.758+0000 7f200ea61700 1 -- start start 2024-03-12T03:14:34.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.758+0000 7f200ea61700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f20081a5910 con 0x7f2008106a70 2024-03-12T03:14:34.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.758+0000 7f200ea61700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f20081a6b30 con 0x7f200810d4f0 2024-03-12T03:14:34.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.758+0000 7f200ea61700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f20081a7d70 con 0x7f2008109920 2024-03-12T03:14:34.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f2007fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2008106a70 0x7f200810ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37264/0 (socket says 172.21.15.94:37264) 2024-03-12T03:14:34.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20077fe700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f2008109920 0x7f20081a09a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:57168/0 (socket says 172.21.15.94:57168) 2024-03-12T03:14:34.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f2007fff700 1 -- 172.21.15.94:0/4002446652 learned_addr learned my addr 172.21.15.94:0/4002446652 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:34.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851171175 0 0) 0x7f20081a7d70 con 0x7f2008109920 2024-03-12T03:14:34.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff00035b0 con 0x7f2008109920 2024-03-12T03:14:34.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036165876 0 0) 0x7f20081a5910 con 0x7f2008106a70 2024-03-12T03:14:34.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20081a7d70 con 0x7f2008106a70 2024-03-12T03:14:34.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230764756 0 0) 0x7f20081a6b30 con 0x7f200810d4f0 2024-03-12T03:14:34.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20081a5910 con 0x7f200810d4f0 2024-03-12T03:14:34.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 534594749 0 0) 0x7f1ff00035b0 con 0x7f2008109920 2024-03-12T03:14:34.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20081a6b30 con 0x7f2008109920 2024-03-12T03:14:34.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2170682386 0 0) 0x7f20081a7d70 con 0x7f2008106a70 2024-03-12T03:14:34.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ff00035b0 con 0x7f2008106a70 2024-03-12T03:14:34.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1ffc0013d0 con 0x7f2008109920 2024-03-12T03:14:34.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1ff80045d0 con 0x7f2008106a70 2024-03-12T03:14:34.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 51932735 0 0) 0x7f20081a5910 con 0x7f200810d4f0 2024-03-12T03:14:34.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20081a7d70 con 0x7f200810d4f0 2024-03-12T03:14:34.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f2000001b30 con 0x7f200810d4f0 2024-03-12T03:14:34.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3884295043 0 0) 0x7f20081a6b30 con 0x7f2008109920 2024-03-12T03:14:34.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 >> v1:172.21.15.151:6789/0 conn(0x7f200810d4f0 legacy=0x7f20081a40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:34.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 >> v1:172.21.15.94:6789/0 conn(0x7f2008106a70 legacy=0x7f200810ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:34.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20081a8fb0 con 0x7f2008109920 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f200ea61700 1 -- 172.21.15.94:0/4002446652 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20081a5b40 con 0x7f2008109920 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.759+0000 7f200ea61700 1 -- 172.21.15.94:0/4002446652 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20081a6000 con 0x7f2008109920 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.760+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f1ffc001630 con 0x7f2008109920 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.760+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1ffc005b80 con 0x7f2008109920 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.760+0000 7f200ea61700 1 -- 172.21.15.94:0/4002446652 --> v1:172.21.15.94:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1ff4000fb0 con 0x7f2008109920 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.761+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f1ffc01c210 con 0x7f2008109920 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.761+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7f1ffc088e90 con 0x7f2008109920 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.761+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 --> v1:172.21.15.151:6800/2666240674 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ff007d090 con 0x7f1ff0079380 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.761+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7f1ffc089360 con 0x7f2008109920 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.763+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== osd.4 v1:172.21.15.151:6800/2666240674 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1ff007d090 con 0x7f1ff0079380 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.779+0000 7f200ea61700 1 -- 172.21.15.94:0/4002446652 --> v1:172.21.15.151:6800/2666240674 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1ff4002d10 con 0x7f1ff0079380 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.780+0000 7f20057fa700 1 -- 172.21.15.94:0/4002446652 <== osd.4 v1:172.21.15.151:6800/2666240674 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 1156102027) 0x7f1ff4002d10 con 0x7f1ff0079380 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.780+0000 7f200ea61700 1 -- 172.21.15.94:0/4002446652 >> v1:172.21.15.151:6800/2666240674 conn(0x7f1ff0079380 legacy=0x7f1ff007b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.780+0000 7f200ea61700 1 -- 172.21.15.94:0/4002446652 >> v1:172.21.15.94:6800/2099808905 conn(0x7f1ff006eca0 legacy=0x7f1ff0071160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:34.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.780+0000 7f200ea61700 1 -- 172.21.15.94:0/4002446652 >> v1:172.21.15.94:6790/0 conn(0x7f2008109920 legacy=0x7f20081a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:34.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.780+0000 7f200cffe700 1 -- 172.21.15.94:0/4002446652 reap_dead start 2024-03-12T03:14:34.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.780+0000 7f200ea61700 1 -- 172.21.15.94:0/4002446652 shutdown_connections 2024-03-12T03:14:34.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.780+0000 7f200ea61700 1 -- 172.21.15.94:0/4002446652 >> 172.21.15.94:0/4002446652 conn(0x7f20080fc580 msgr2=0x7f2008110180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:34.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.780+0000 7f200ea61700 1 -- 172.21.15.94:0/4002446652 shutdown_connections 2024-03-12T03:14:34.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:34.780+0000 7f200ea61700 1 -- 172.21.15.94:0/4002446652 wait complete. 2024-03-12T03:14:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:34 smithi151 ceph-mon[32923]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:14:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:35.142 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:35.146 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:35.783 INFO:teuthology.orchestra.run.smithi094.stdout:128849018910 2024-03-12T03:14:35.783 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-12T03:14:36.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:36 smithi094 ceph-mon[29291]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:36.893 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:36 smithi094 ceph-mon[40484]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:36 smithi151 ceph-mon[32923]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:37.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:14:37 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:14:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:38 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:38 smithi094 ceph-mon[29291]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:14:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:38 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:38 smithi094 ceph-mon[40484]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:14:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:38 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:38 smithi151 ceph-mon[32923]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:14:40.794 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:40 smithi094 ceph-mon[40484]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:40.795 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:14:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:14:40.795 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:40 smithi094 ceph-mon[29291]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:40 smithi151 ceph-mon[32923]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:42.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.269+0000 7f45b6614700 1 -- 172.21.15.94:0/1448332456 >> v1:172.21.15.94:6789/0 conn(0x7f45b010e6d0 legacy=0x7f45b0110b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:42.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.269+0000 7f45b6614700 1 -- 172.21.15.94:0/1448332456 shutdown_connections 2024-03-12T03:14:42.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.269+0000 7f45b6614700 1 -- 172.21.15.94:0/1448332456 >> 172.21.15.94:0/1448332456 conn(0x7f45b0075d70 msgr2=0x7f45b0076180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:42.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.269+0000 7f45b6614700 1 -- 172.21.15.94:0/1448332456 shutdown_connections 2024-03-12T03:14:42.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.269+0000 7f45b6614700 1 -- 172.21.15.94:0/1448332456 wait complete. 2024-03-12T03:14:42.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.269+0000 7f45b6614700 1 Processor -- start 2024-03-12T03:14:42.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.269+0000 7f45b6614700 1 -- start start 2024-03-12T03:14:42.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45b6614700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f45b01a6de0 con 0x7f45b010aa30 2024-03-12T03:14:42.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45b6614700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f45b01a8000 con 0x7f45b0106c00 2024-03-12T03:14:42.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45b6614700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f45b01a9240 con 0x7f45b010e6d0 2024-03-12T03:14:42.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45b4bb1700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f45b010e6d0 0x7f45b01a54a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:57180/0 (socket says 172.21.15.94:57180) 2024-03-12T03:14:42.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45b4bb1700 1 -- 172.21.15.94:0/3527450200 learned_addr learned my addr 172.21.15.94:0/3527450200 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:42.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069890155 0 0) 0x7f45b01a6de0 con 0x7f45b010aa30 2024-03-12T03:14:42.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45980035b0 con 0x7f45b010aa30 2024-03-12T03:14:42.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430730914 0 0) 0x7f45b01a9240 con 0x7f45b010e6d0 2024-03-12T03:14:42.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b01a6de0 con 0x7f45b010e6d0 2024-03-12T03:14:42.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119569189 0 0) 0x7f45b01a8000 con 0x7f45b0106c00 2024-03-12T03:14:42.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b01a9240 con 0x7f45b0106c00 2024-03-12T03:14:42.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1217455444 0 0) 0x7f45980035b0 con 0x7f45b010aa30 2024-03-12T03:14:42.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45b01a8000 con 0x7f45b010aa30 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f45a40018d0 con 0x7f45b010aa30 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3492566184 0 0) 0x7f45b01a6de0 con 0x7f45b010e6d0 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45980035b0 con 0x7f45b010e6d0 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.270+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f45a8001c00 con 0x7f45b010e6d0 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.271+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 928233292 0 0) 0x7f45b01a8000 con 0x7f45b010aa30 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.271+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 >> v1:172.21.15.94:6790/0 conn(0x7f45b010e6d0 legacy=0x7f45b01a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.271+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 >> v1:172.21.15.151:6789/0 conn(0x7f45b0106c00 legacy=0x7f45b0100d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.271+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b01aa480 con 0x7f45b010aa30 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.271+0000 7f45b6614700 1 -- 172.21.15.94:0/3527450200 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b01a9470 con 0x7f45b010aa30 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.271+0000 7f45b6614700 1 -- 172.21.15.94:0/3527450200 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b01a9930 con 0x7f45b010aa30 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.271+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f45a4003b30 con 0x7f45b010aa30 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.271+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f45a4004ac0 con 0x7f45b010aa30 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.271+0000 7f45b6614700 1 -- 172.21.15.94:0/3527450200 --> v1:172.21.15.94:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f459c000f80 con 0x7f45b010aa30 2024-03-12T03:14:42.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.272+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f45a401b130 con 0x7f45b010aa30 2024-03-12T03:14:42.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.272+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7f45a4087f30 con 0x7f45b010aa30 2024-03-12T03:14:42.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.273+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 --> v1:172.21.15.151:6804/2718612978 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f459807cdd0 con 0x7f45980790c0 2024-03-12T03:14:42.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.273+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7f45a40882c0 con 0x7f45b010aa30 2024-03-12T03:14:42.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.274+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== osd.5 v1:172.21.15.151:6804/2718612978 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f459807cdd0 con 0x7f45980790c0 2024-03-12T03:14:42.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.290+0000 7f45b6614700 1 -- 172.21.15.94:0/3527450200 --> v1:172.21.15.151:6804/2718612978 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f459c002ce0 con 0x7f45980790c0 2024-03-12T03:14:42.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.291+0000 7f45ad7fa700 1 -- 172.21.15.94:0/3527450200 <== osd.5 v1:172.21.15.151:6804/2718612978 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 3852841860) 0x7f459c002ce0 con 0x7f45980790c0 2024-03-12T03:14:42.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.291+0000 7f45b6614700 1 -- 172.21.15.94:0/3527450200 >> v1:172.21.15.151:6804/2718612978 conn(0x7f45980790c0 legacy=0x7f459807b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:42.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.292+0000 7f45b6614700 1 -- 172.21.15.94:0/3527450200 >> v1:172.21.15.94:6800/2099808905 conn(0x7f459806e9e0 legacy=0x7f4598070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:42.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.292+0000 7f45b6614700 1 -- 172.21.15.94:0/3527450200 >> v1:172.21.15.94:6789/0 conn(0x7f45b010aa30 legacy=0x7f45b01a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:42.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.292+0000 7f45b4bb1700 1 -- 172.21.15.94:0/3527450200 reap_dead start 2024-03-12T03:14:42.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.292+0000 7f45b6614700 1 -- 172.21.15.94:0/3527450200 shutdown_connections 2024-03-12T03:14:42.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.292+0000 7f45b6614700 1 -- 172.21.15.94:0/3527450200 >> 172.21.15.94:0/3527450200 conn(0x7f45b0075d70 msgr2=0x7f45b0111bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:42.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.292+0000 7f45b6614700 1 -- 172.21.15.94:0/3527450200 shutdown_connections 2024-03-12T03:14:42.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:42.292+0000 7f45b6614700 1 -- 172.21.15.94:0/3527450200 wait complete. 2024-03-12T03:14:42.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:42 smithi094 ceph-mon[29291]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:42 smithi094 ceph-mon[40484]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:42 smithi151 ceph-mon[32923]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:43.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.748+0000 7f8c351e1700 1 -- 172.21.15.94:0/2677196024 >> v1:172.21.15.94:6789/0 conn(0x7f8c30109920 legacy=0x7f8c3010bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:43.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.748+0000 7f8c351e1700 1 -- 172.21.15.94:0/2677196024 shutdown_connections 2024-03-12T03:14:43.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.748+0000 7f8c351e1700 1 -- 172.21.15.94:0/2677196024 >> 172.21.15.94:0/2677196024 conn(0x7f8c300fc580 msgr2=0x7f8c300fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:43.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.748+0000 7f8c351e1700 1 -- 172.21.15.94:0/2677196024 shutdown_connections 2024-03-12T03:14:43.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.748+0000 7f8c351e1700 1 -- 172.21.15.94:0/2677196024 wait complete. 2024-03-12T03:14:43.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.749+0000 7f8c351e1700 1 Processor -- start 2024-03-12T03:14:43.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.749+0000 7f8c351e1700 1 -- start start 2024-03-12T03:14:43.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c351e1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8c3010f6d0 con 0x7f8c30109920 2024-03-12T03:14:43.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c351e1700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8c3010f8e0 con 0x7f8c30106a70 2024-03-12T03:14:43.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c351e1700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8c301ab510 con 0x7f8c3010d4f0 2024-03-12T03:14:43.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c2f59e700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f8c3010d4f0 0x7f8c301a7dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:57210/0 (socket says 172.21.15.94:57210) 2024-03-12T03:14:43.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c2e59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8c30109920 0x7f8c3010e180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37302/0 (socket says 172.21.15.94:37302) 2024-03-12T03:14:43.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c2f59e700 1 -- 172.21.15.94:0/582742494 learned_addr learned my addr 172.21.15.94:0/582742494 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:43.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220795990 0 0) 0x7f8c3010f6d0 con 0x7f8c30109920 2024-03-12T03:14:43.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c180035b0 con 0x7f8c30109920 2024-03-12T03:14:43.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300087681 0 0) 0x7f8c301ab510 con 0x7f8c3010d4f0 2024-03-12T03:14:43.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c3010f6d0 con 0x7f8c3010d4f0 2024-03-12T03:14:43.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215961072 0 0) 0x7f8c3010f8e0 con 0x7f8c30106a70 2024-03-12T03:14:43.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c301ab510 con 0x7f8c30106a70 2024-03-12T03:14:43.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2102954032 0 0) 0x7f8c180035b0 con 0x7f8c30109920 2024-03-12T03:14:43.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c3010f8e0 con 0x7f8c30109920 2024-03-12T03:14:43.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8c24001a20 con 0x7f8c30109920 2024-03-12T03:14:43.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3111847033 0 0) 0x7f8c301ab510 con 0x7f8c30106a70 2024-03-12T03:14:43.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c180035b0 con 0x7f8c30106a70 2024-03-12T03:14:43.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.750+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8c20001830 con 0x7f8c30106a70 2024-03-12T03:14:43.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.751+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 382413337 0 0) 0x7f8c3010f6d0 con 0x7f8c3010d4f0 2024-03-12T03:14:43.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.751+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c301ab510 con 0x7f8c3010d4f0 2024-03-12T03:14:43.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.751+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8c28001c50 con 0x7f8c3010d4f0 2024-03-12T03:14:43.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.751+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3527493809 0 0) 0x7f8c180035b0 con 0x7f8c30106a70 2024-03-12T03:14:43.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.751+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 >> v1:172.21.15.94:6790/0 conn(0x7f8c3010d4f0 legacy=0x7f8c301a7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:43.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.751+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 >> v1:172.21.15.94:6789/0 conn(0x7f8c30109920 legacy=0x7f8c3010e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:43.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.751+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c301ac6f0 con 0x7f8c30106a70 2024-03-12T03:14:43.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.751+0000 7f8c351e1700 1 -- 172.21.15.94:0/582742494 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c301ab740 con 0x7f8c30106a70 2024-03-12T03:14:43.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.751+0000 7f8c351e1700 1 -- 172.21.15.94:0/582742494 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c301abc00 con 0x7f8c30106a70 2024-03-12T03:14:43.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.751+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f8c20003b10 con 0x7f8c30106a70 2024-03-12T03:14:43.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.751+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8c20004a40 con 0x7f8c30106a70 2024-03-12T03:14:43.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.752+0000 7f8c351e1700 1 -- 172.21.15.94:0/582742494 --> v1:172.21.15.151:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8c1c000f80 con 0x7f8c30106a70 2024-03-12T03:14:43.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.753+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f8c20003680 con 0x7f8c30106a70 2024-03-12T03:14:43.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.754+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7f8c20087fa0 con 0x7f8c30106a70 2024-03-12T03:14:43.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.754+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 --> v1:172.21.15.94:6809/2929979688 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c1807d1f0 con 0x7f8c180794e0 2024-03-12T03:14:43.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.754+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7f8c20088330 con 0x7f8c30106a70 2024-03-12T03:14:43.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.755+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== osd.2 v1:172.21.15.94:6809/2929979688 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8c1807d1f0 con 0x7f8c180794e0 2024-03-12T03:14:43.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.777+0000 7f8c351e1700 1 -- 172.21.15.94:0/582742494 --> v1:172.21.15.94:6809/2929979688 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8c1c002ce0 con 0x7f8c180794e0 2024-03-12T03:14:43.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.778+0000 7f8c17fff700 1 -- 172.21.15.94:0/582742494 <== osd.2 v1:172.21.15.94:6809/2929979688 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1468241469) 0x7f8c1c002ce0 con 0x7f8c180794e0 2024-03-12T03:14:43.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.778+0000 7f8c351e1700 1 -- 172.21.15.94:0/582742494 >> v1:172.21.15.94:6809/2929979688 conn(0x7f8c180794e0 legacy=0x7f8c1807b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:43.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.778+0000 7f8c351e1700 1 -- 172.21.15.94:0/582742494 >> v1:172.21.15.94:6800/2099808905 conn(0x7f8c1806eda0 legacy=0x7f8c18071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:43.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.778+0000 7f8c351e1700 1 -- 172.21.15.94:0/582742494 >> v1:172.21.15.151:6789/0 conn(0x7f8c30106a70 legacy=0x7f8c3010da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:43.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.778+0000 7f8c2f59e700 1 -- 172.21.15.94:0/582742494 reap_dead start 2024-03-12T03:14:43.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.778+0000 7f8c351e1700 1 -- 172.21.15.94:0/582742494 shutdown_connections 2024-03-12T03:14:43.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.778+0000 7f8c351e1700 1 -- 172.21.15.94:0/582742494 >> 172.21.15.94:0/582742494 conn(0x7f8c300fc580 msgr2=0x7f8c30100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:43.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.778+0000 7f8c351e1700 1 -- 172.21.15.94:0/582742494 shutdown_connections 2024-03-12T03:14:43.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:43.778+0000 7f8c351e1700 1 -- 172.21.15.94:0/582742494 wait complete. 2024-03-12T03:14:44.934 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:44 smithi094 ceph-mon[29291]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:44.935 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:44 smithi094 ceph-mon[40484]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:44 smithi151 ceph-mon[32923]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:45.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.665+0000 7fb7da22f700 1 -- 172.21.15.94:0/1442428062 >> v1:172.21.15.94:6789/0 conn(0x7fb7d410e640 legacy=0x7fb7d4110b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.666+0000 7fb7da22f700 1 -- 172.21.15.94:0/1442428062 shutdown_connections 2024-03-12T03:14:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.666+0000 7fb7da22f700 1 -- 172.21.15.94:0/1442428062 >> 172.21.15.94:0/1442428062 conn(0x7fb7d4075dd0 msgr2=0x7fb7d40761e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.666+0000 7fb7da22f700 1 -- 172.21.15.94:0/1442428062 shutdown_connections 2024-03-12T03:14:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.666+0000 7fb7da22f700 1 -- 172.21.15.94:0/1442428062 wait complete. 2024-03-12T03:14:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.666+0000 7fb7da22f700 1 Processor -- start 2024-03-12T03:14:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.666+0000 7fb7da22f700 1 -- start start 2024-03-12T03:14:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.666+0000 7fb7da22f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb7d41a6d90 con 0x7fb7d4107cd0 2024-03-12T03:14:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7da22f700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb7d41a7fb0 con 0x7fb7d410e640 2024-03-12T03:14:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7da22f700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb7d41a91f0 con 0x7fb7d410aa40 2024-03-12T03:14:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d2ffd700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fb7d410aa40 0x7fb7d41a1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48310/0 (socket says 172.21.15.94:48310) 2024-03-12T03:14:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d2ffd700 1 -- 172.21.15.94:0/4288600942 learned_addr learned my addr 172.21.15.94:0/4288600942 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210404798 0 0) 0x7fb7d41a91f0 con 0x7fb7d410aa40 2024-03-12T03:14:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7bc0035b0 con 0x7fb7d410aa40 2024-03-12T03:14:45.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633688943 0 0) 0x7fb7d41a6d90 con 0x7fb7d4107cd0 2024-03-12T03:14:45.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d41a91f0 con 0x7fb7d4107cd0 2024-03-12T03:14:45.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333530313 0 0) 0x7fb7d41a7fb0 con 0x7fb7d410e640 2024-03-12T03:14:45.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d41a6d90 con 0x7fb7d410e640 2024-03-12T03:14:45.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 228919418 0 0) 0x7fb7bc0035b0 con 0x7fb7d410aa40 2024-03-12T03:14:45.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7d41a7fb0 con 0x7fb7d410aa40 2024-03-12T03:14:45.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fb7c8001970 con 0x7fb7d410aa40 2024-03-12T03:14:45.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1208964253 0 0) 0x7fb7d41a6d90 con 0x7fb7d410e640 2024-03-12T03:14:45.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7bc0035b0 con 0x7fb7d410e640 2024-03-12T03:14:45.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fb7cc001b70 con 0x7fb7d410e640 2024-03-12T03:14:45.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1539965843 0 0) 0x7fb7d41a7fb0 con 0x7fb7d410aa40 2024-03-12T03:14:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 >> v1:172.21.15.151:6789/0 conn(0x7fb7d410e640 legacy=0x7fb7d41a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 >> v1:172.21.15.94:6789/0 conn(0x7fb7d4107cd0 legacy=0x7fb7d4107130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d41aa430 con 0x7fb7d410aa40 2024-03-12T03:14:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7da22f700 1 -- 172.21.15.94:0/4288600942 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d41a9420 con 0x7fb7d410aa40 2024-03-12T03:14:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.667+0000 7fb7da22f700 1 -- 172.21.15.94:0/4288600942 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d41a98e0 con 0x7fb7d410aa40 2024-03-12T03:14:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.668+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fb7c8001d20 con 0x7fb7d410aa40 2024-03-12T03:14:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.668+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fb7c8005d50 con 0x7fb7d410aa40 2024-03-12T03:14:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.668+0000 7fb7da22f700 1 -- 172.21.15.94:0/4288600942 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c0005120 con 0x7fb7d410aa40 2024-03-12T03:14:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.669+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7fb7c801c640 con 0x7fb7d410aa40 2024-03-12T03:14:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.669+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7fb7c8089080 con 0x7fb7d410aa40 2024-03-12T03:14:45.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.674+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fb7c8056a00 con 0x7fb7d410aa40 2024-03-12T03:14:45.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.880+0000 7fb7da22f700 1 -- 172.21.15.94:0/4288600942 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb7c0005410 con 0x7fb7d410aa40 2024-03-12T03:14:45.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.880+0000 7fb7d0ff9700 1 -- 172.21.15.94:0/4288600942 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (unknown 1482059429 0 703582123) 0x7fb7c805a450 con 0x7fb7d410aa40 2024-03-12T03:14:45.883 INFO:teuthology.orchestra.run.smithi094.stdout:231928234001 2024-03-12T03:14:45.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.882+0000 7fb7da22f700 1 -- 172.21.15.94:0/4288600942 >> v1:172.21.15.94:6800/2099808905 conn(0x7fb7bc06e870 legacy=0x7fb7bc070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:45.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.882+0000 7fb7da22f700 1 -- 172.21.15.94:0/4288600942 >> v1:172.21.15.94:6790/0 conn(0x7fb7d410aa40 legacy=0x7fb7d41a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:45.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.882+0000 7fb7da22f700 1 -- 172.21.15.94:0/4288600942 shutdown_connections 2024-03-12T03:14:45.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.882+0000 7fb7da22f700 1 -- 172.21.15.94:0/4288600942 >> 172.21.15.94:0/4288600942 conn(0x7fb7d4075dd0 msgr2=0x7fb7d41092e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:45.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.882+0000 7fb7da22f700 1 -- 172.21.15.94:0/4288600942 shutdown_connections 2024-03-12T03:14:45.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:45.883+0000 7fb7da22f700 1 -- 172.21.15.94:0/4288600942 wait complete. 2024-03-12T03:14:46.190 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4288600942' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T03:14:46.190 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4288600942' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T03:14:46.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4288600942' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T03:14:46.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.288+0000 7fc662b41700 1 -- 172.21.15.94:0/3631721721 >> v1:172.21.15.94:6790/0 conn(0x7fc65c101df0 legacy=0x7fc65c1021d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:46.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.288+0000 7fc662b41700 1 -- 172.21.15.94:0/3631721721 shutdown_connections 2024-03-12T03:14:46.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.288+0000 7fc662b41700 1 -- 172.21.15.94:0/3631721721 >> 172.21.15.94:0/3631721721 conn(0x7fc65c0fd680 msgr2=0x7fc65c0ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:46.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.288+0000 7fc662b41700 1 -- 172.21.15.94:0/3631721721 shutdown_connections 2024-03-12T03:14:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.288+0000 7fc662b41700 1 -- 172.21.15.94:0/3631721721 wait complete. 2024-03-12T03:14:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.288+0000 7fc662b41700 1 Processor -- start 2024-03-12T03:14:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc662b41700 1 -- start start 2024-03-12T03:14:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc662b41700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc65c1a6b80 con 0x7fc65c10aa80 2024-03-12T03:14:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc662b41700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc65c1a7da0 con 0x7fc65c10e620 2024-03-12T03:14:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc662b41700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc65c1a8fe0 con 0x7fc65c101df0 2024-03-12T03:14:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc6608dd700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fc65c101df0 0x7fc65c107020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48330/0 (socket says 172.21.15.94:48330) 2024-03-12T03:14:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc6608dd700 1 -- 172.21.15.94:0/1781699147 learned_addr learned my addr 172.21.15.94:0/1781699147 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:46.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219525346 0 0) 0x7fc65c1a8fe0 con 0x7fc65c101df0 2024-03-12T03:14:46.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc64c0031f0 con 0x7fc65c101df0 2024-03-12T03:14:46.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228725585 0 0) 0x7fc65c1a6b80 con 0x7fc65c10aa80 2024-03-12T03:14:46.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc65c1a8fe0 con 0x7fc65c10aa80 2024-03-12T03:14:46.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408199584 0 0) 0x7fc65c1a7da0 con 0x7fc65c10e620 2024-03-12T03:14:46.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc65c1a6b80 con 0x7fc65c10e620 2024-03-12T03:14:46.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 41104115 0 0) 0x7fc64c0031f0 con 0x7fc65c101df0 2024-03-12T03:14:46.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc65c1a7da0 con 0x7fc65c101df0 2024-03-12T03:14:46.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc650004390 con 0x7fc65c101df0 2024-03-12T03:14:46.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 52562165 0 0) 0x7fc65c1a8fe0 con 0x7fc65c10aa80 2024-03-12T03:14:46.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc64c0031f0 con 0x7fc65c10aa80 2024-03-12T03:14:46.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc6480017c0 con 0x7fc65c10aa80 2024-03-12T03:14:46.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 916920548 0 0) 0x7fc65c1a6b80 con 0x7fc65c10e620 2024-03-12T03:14:46.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.289+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc65c1a8fe0 con 0x7fc65c10e620 2024-03-12T03:14:46.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.290+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc654001af0 con 0x7fc65c10e620 2024-03-12T03:14:46.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.290+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 681807463 0 0) 0x7fc65c1a7da0 con 0x7fc65c101df0 2024-03-12T03:14:46.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.290+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 >> v1:172.21.15.151:6789/0 conn(0x7fc65c10e620 legacy=0x7fc65c1a5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:46.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.290+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 >> v1:172.21.15.94:6789/0 conn(0x7fc65c10aa80 legacy=0x7fc65c1a1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:46.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.290+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c1aa220 con 0x7fc65c101df0 2024-03-12T03:14:46.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.290+0000 7fc662b41700 1 -- 172.21.15.94:0/1781699147 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65c1a6db0 con 0x7fc65c101df0 2024-03-12T03:14:46.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.290+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fc650001a30 con 0x7fc65c101df0 2024-03-12T03:14:46.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.290+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc650004ff0 con 0x7fc65c101df0 2024-03-12T03:14:46.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.290+0000 7fc662b41700 1 -- 172.21.15.94:0/1781699147 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65c1a7270 con 0x7fc65c101df0 2024-03-12T03:14:46.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.291+0000 7fc662b41700 1 -- 172.21.15.94:0/1781699147 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65c1032d0 con 0x7fc65c101df0 2024-03-12T03:14:46.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.291+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7fc6500039f0 con 0x7fc65c101df0 2024-03-12T03:14:46.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.292+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7fc650089200 con 0x7fc65c101df0 2024-03-12T03:14:46.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.296+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fc650056a40 con 0x7fc65c101df0 2024-03-12T03:14:46.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.480+0000 7f1c09f2a700 1 -- 172.21.15.94:0/134001419 >> v1:172.21.15.94:6789/0 conn(0x7f1c04109920 legacy=0x7f1c0410bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:46.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.480+0000 7f1c09f2a700 1 -- 172.21.15.94:0/134001419 shutdown_connections 2024-03-12T03:14:46.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.480+0000 7f1c09f2a700 1 -- 172.21.15.94:0/134001419 >> 172.21.15.94:0/134001419 conn(0x7f1c040fc580 msgr2=0x7f1c040fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:46.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.480+0000 7f1c09f2a700 1 -- 172.21.15.94:0/134001419 shutdown_connections 2024-03-12T03:14:46.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.480+0000 7f1c09f2a700 1 -- 172.21.15.94:0/134001419 wait complete. 2024-03-12T03:14:46.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.481+0000 7f1c09f2a700 1 Processor -- start 2024-03-12T03:14:46.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.481+0000 7f1c09f2a700 1 -- start start 2024-03-12T03:14:46.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.481+0000 7f1c09f2a700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1c041a5c60 con 0x7f1c0410d4f0 2024-03-12T03:14:46.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.481+0000 7f1c09f2a700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1c041a6e80 con 0x7f1c04109920 2024-03-12T03:14:46.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.481+0000 7f1c09f2a700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1c041a80c0 con 0x7f1c04106a70 2024-03-12T03:14:46.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.481+0000 7f1c037fe700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f1c04106a70 0x7f1c0410cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48358/0 (socket says 172.21.15.94:48358) 2024-03-12T03:14:46.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.481+0000 7f1c037fe700 1 -- 172.21.15.94:0/3436102765 learned_addr learned my addr 172.21.15.94:0/3436102765 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:46.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.481+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040683855 0 0) 0x7f1c041a80c0 con 0x7f1c04106a70 2024-03-12T03:14:46.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bec0035b0 con 0x7f1c04106a70 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464413380 0 0) 0x7f1c041a5c60 con 0x7f1c0410d4f0 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c041a80c0 con 0x7f1c0410d4f0 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111714345 0 0) 0x7f1c041a6e80 con 0x7f1c04109920 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c041a5c60 con 0x7f1c04109920 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4039448872 0 0) 0x7f1bec0035b0 con 0x7f1c04106a70 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c041a6e80 con 0x7f1c04106a70 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1855532073 0 0) 0x7f1c041a80c0 con 0x7f1c0410d4f0 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bec0035b0 con 0x7f1c0410d4f0 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1bf40018b0 con 0x7f1c04106a70 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1bfc001bb0 con 0x7f1c0410d4f0 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3692087078 0 0) 0x7f1c041a5c60 con 0x7f1c04109920 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c041a80c0 con 0x7f1c04109920 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 375490730 0 0) 0x7f1c041a6e80 con 0x7f1c04106a70 2024-03-12T03:14:46.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 >> v1:172.21.15.151:6789/0 conn(0x7f1c04109920 legacy=0x7f1c041a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:46.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 >> v1:172.21.15.94:6789/0 conn(0x7f1c0410d4f0 legacy=0x7f1c041a4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:46.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c041a9300 con 0x7f1c04106a70 2024-03-12T03:14:46.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c09f2a700 1 -- 172.21.15.94:0/3436102765 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c041a5e90 con 0x7f1c04106a70 2024-03-12T03:14:46.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c09f2a700 1 -- 172.21.15.94:0/3436102765 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c041a6350 con 0x7f1c04106a70 2024-03-12T03:14:46.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f1bf4003b30 con 0x7f1c04106a70 2024-03-12T03:14:46.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.482+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1bf4005cf0 con 0x7f1c04106a70 2024-03-12T03:14:46.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.483+0000 7f1c09f2a700 1 -- 172.21.15.94:0/3436102765 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf0005150 con 0x7f1c04106a70 2024-03-12T03:14:46.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.484+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f1bf4006fa0 con 0x7f1c04106a70 2024-03-12T03:14:46.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.484+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7f1bf4088fe0 con 0x7f1c04106a70 2024-03-12T03:14:46.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.489+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f1bf4056880 con 0x7f1c04106a70 2024-03-12T03:14:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.498+0000 7fc662b41700 1 -- 172.21.15.94:0/1781699147 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc65c105600 con 0x7fc65c101df0 2024-03-12T03:14:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.499+0000 7fc659ffb700 1 -- 172.21.15.94:0/1781699147 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (unknown 1274345170 0 80793029) 0x7fc650050e20 con 0x7fc65c101df0 2024-03-12T03:14:46.502 INFO:teuthology.orchestra.run.smithi094.stdout:197568495638 2024-03-12T03:14:46.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.501+0000 7fc662b41700 1 -- 172.21.15.94:0/1781699147 >> v1:172.21.15.94:6800/2099808905 conn(0x7fc64c06ed80 legacy=0x7fc64c071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:46.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.501+0000 7fc662b41700 1 -- 172.21.15.94:0/1781699147 >> v1:172.21.15.94:6790/0 conn(0x7fc65c101df0 legacy=0x7fc65c107020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:46.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.501+0000 7fc662b41700 1 -- 172.21.15.94:0/1781699147 shutdown_connections 2024-03-12T03:14:46.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.501+0000 7fc662b41700 1 -- 172.21.15.94:0/1781699147 >> 172.21.15.94:0/1781699147 conn(0x7fc65c0fd680 msgr2=0x7fc65c107950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:46.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.501+0000 7fc662b41700 1 -- 172.21.15.94:0/1781699147 shutdown_connections 2024-03-12T03:14:46.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.501+0000 7fc662b41700 1 -- 172.21.15.94:0/1781699147 wait complete. 2024-03-12T03:14:46.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.693+0000 7f1c09f2a700 1 -- 172.21.15.94:0/3436102765 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1bf0005440 con 0x7f1c04106a70 2024-03-12T03:14:46.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.693+0000 7f1c00ff9700 1 -- 172.21.15.94:0/3436102765 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (unknown 383520633 0 1575966472) 0x7f1bf405a2d0 con 0x7f1c04106a70 2024-03-12T03:14:46.696 INFO:teuthology.orchestra.run.smithi094.stdout:107374182436 2024-03-12T03:14:46.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.695+0000 7f1c09f2a700 1 -- 172.21.15.94:0/3436102765 >> v1:172.21.15.94:6800/2099808905 conn(0x7f1bec06ed40 legacy=0x7f1bec071200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:46.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.695+0000 7f1c09f2a700 1 -- 172.21.15.94:0/3436102765 >> v1:172.21.15.94:6790/0 conn(0x7f1c04106a70 legacy=0x7f1c0410cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:46.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.695+0000 7f1c09f2a700 1 -- 172.21.15.94:0/3436102765 shutdown_connections 2024-03-12T03:14:46.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.695+0000 7f1c09f2a700 1 -- 172.21.15.94:0/3436102765 >> 172.21.15.94:0/3436102765 conn(0x7f1c040fc580 msgr2=0x7f1c04110910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:46.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.696+0000 7f1c09f2a700 1 -- 172.21.15.94:0/3436102765 shutdown_connections 2024-03-12T03:14:46.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.696+0000 7f1c09f2a700 1 -- 172.21.15.94:0/3436102765 wait complete. 2024-03-12T03:14:47.039 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1781699147' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T03:14:47.039 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:46 smithi094 ceph-mon[29291]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:47.039 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:46 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:14:47.039 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3436102765' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:14:47.039 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1781699147' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T03:14:47.039 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:46 smithi094 ceph-mon[40484]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:47.040 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:46 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:14:47.040 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3436102765' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:14:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1781699147' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T03:14:47.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:46 smithi151 ceph-mon[32923]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:47.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:46 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:14:47.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3436102765' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:14:47.680 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495634 got 197568495638 for osd.6 2024-03-12T03:14:47.680 DEBUG:teuthology.parallel:result is None 2024-03-12T03:14:47.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.823+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2228161997 >> v1:172.21.15.94:6790/0 conn(0x7fc6a010d500 legacy=0x7fc6a010f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:47.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.823+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2228161997 shutdown_connections 2024-03-12T03:14:47.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.823+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2228161997 >> 172.21.15.94:0/2228161997 conn(0x7fc6a00fc590 msgr2=0x7fc6a00fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:47.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.824+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2228161997 shutdown_connections 2024-03-12T03:14:47.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.824+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2228161997 wait complete. 2024-03-12T03:14:47.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.824+0000 7fc6a59a1700 1 Processor -- start 2024-03-12T03:14:47.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.824+0000 7fc6a59a1700 1 -- start start 2024-03-12T03:14:47.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.824+0000 7fc6a59a1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6a01a6bc0 con 0x7fc6a0106a80 2024-03-12T03:14:47.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.824+0000 7fc6a59a1700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6a01a7e00 con 0x7fc6a0109930 2024-03-12T03:14:47.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.824+0000 7fc6a59a1700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6a01a9040 con 0x7fc6a010d500 2024-03-12T03:14:47.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc69effd700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc6a0106a80 0x7fc6a010ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52968/0 (socket says 172.21.15.94:52968) 2024-03-12T03:14:47.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc69f7fe700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fc6a010d500 0x7fc6a01a52c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48376/0 (socket says 172.21.15.94:48376) 2024-03-12T03:14:47.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc69f7fe700 1 -- 172.21.15.94:0/2388606824 learned_addr learned my addr 172.21.15.94:0/2388606824 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:47.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276594395 0 0) 0x7fc6a01a9040 con 0x7fc6a010d500 2024-03-12T03:14:47.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6880031f0 con 0x7fc6a010d500 2024-03-12T03:14:47.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524160150 0 0) 0x7fc6a01a6bc0 con 0x7fc6a0106a80 2024-03-12T03:14:47.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a01a9040 con 0x7fc6a0106a80 2024-03-12T03:14:47.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827533071 0 0) 0x7fc6a01a7e00 con 0x7fc6a0109930 2024-03-12T03:14:47.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a01a6bc0 con 0x7fc6a0109930 2024-03-12T03:14:47.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2191844673 0 0) 0x7fc6880031f0 con 0x7fc6a010d500 2024-03-12T03:14:47.759 INFO:teuthology.orchestra.run.smithi094.stdout:55834574895 2024-03-12T03:14:47.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6a01a7e00 con 0x7fc6a010d500 2024-03-12T03:14:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc698001400 con 0x7fc6a010d500 2024-03-12T03:14:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1057598477 0 0) 0x7fc6a01a9040 con 0x7fc6a0106a80 2024-03-12T03:14:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6880031f0 con 0x7fc6a0106a80 2024-03-12T03:14:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc6900017c0 con 0x7fc6a0106a80 2024-03-12T03:14:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1469546418 0 0) 0x7fc6a01a6bc0 con 0x7fc6a0109930 2024-03-12T03:14:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6a01a9040 con 0x7fc6a0109930 2024-03-12T03:14:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc6940017f0 con 0x7fc6a0109930 2024-03-12T03:14:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3409731411 0 0) 0x7fc6a01a7e00 con 0x7fc6a010d500 2024-03-12T03:14:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 >> v1:172.21.15.151:6789/0 conn(0x7fc6a0109930 legacy=0x7fc6a01a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 >> v1:172.21.15.94:6789/0 conn(0x7fc6a0106a80 legacy=0x7fc6a010ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a01aa280 con 0x7fc6a010d500 2024-03-12T03:14:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2388606824 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a01a6df0 con 0x7fc6a010d500 2024-03-12T03:14:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2388606824 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a01a72b0 con 0x7fc6a010d500 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fc698004960 con 0x7fc6a010d500 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.825+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fc698005070 con 0x7fc6a010d500 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.826+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2388606824 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc68c005150 con 0x7fc6a010d500 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.827+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7fc6980052f0 con 0x7fc6a010d500 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.827+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7fc6980895c0 con 0x7fc6a010d500 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:46.832+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fc698056e60 con 0x7fc6a010d500 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:47.036+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2388606824 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc68c005440 con 0x7fc6a010d500 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:47.036+0000 7fc6a499f700 1 -- 172.21.15.94:0/2388606824 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 1791578311) 0x7fc69805a8b0 con 0x7fc6a010d500 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:47.038+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2388606824 >> v1:172.21.15.94:6800/2099808905 conn(0x7fc68806eda0 legacy=0x7fc688071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:47.038+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2388606824 >> v1:172.21.15.94:6790/0 conn(0x7fc6a010d500 legacy=0x7fc6a01a52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:47.038+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2388606824 shutdown_connections 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:47.038+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2388606824 >> 172.21.15.94:0/2388606824 conn(0x7fc6a00fc590 msgr2=0x7fc6a00ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:47.038+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2388606824 shutdown_connections 2024-03-12T03:14:47.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:47.038+0000 7fc6a59a1700 1 -- 172.21.15.94:0/2388606824 wait complete. 2024-03-12T03:14:47.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:14:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:14:48.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2388606824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:14:48.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2388606824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:14:48.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2388606824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:14:48.353 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233997 got 231928234001 for osd.7 2024-03-12T03:14:48.353 DEBUG:teuthology.parallel:result is None 2024-03-12T03:14:48.981 INFO:teuthology.orchestra.run.smithi094.stdout:77309411371 2024-03-12T03:14:48.981 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-12T03:14:49.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:48 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:48 smithi094 ceph-mon[29291]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:48 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:48 smithi094 ceph-mon[40484]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:48 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:48 smithi151 ceph-mon[32923]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:50.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.301+0000 7f9d58d5e700 1 -- 172.21.15.94:0/410400109 >> v1:172.21.15.94:6790/0 conn(0x7f9d5410e6f0 legacy=0x7f9d54110ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.301+0000 7f9d58d5e700 1 -- 172.21.15.94:0/410400109 shutdown_connections 2024-03-12T03:14:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.301+0000 7f9d58d5e700 1 -- 172.21.15.94:0/410400109 >> 172.21.15.94:0/410400109 conn(0x7f9d540fc580 msgr2=0x7f9d540fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.301+0000 7f9d58d5e700 1 -- 172.21.15.94:0/410400109 shutdown_connections 2024-03-12T03:14:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.301+0000 7f9d58d5e700 1 -- 172.21.15.94:0/410400109 wait complete. 2024-03-12T03:14:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.302+0000 7f9d58d5e700 1 Processor -- start 2024-03-12T03:14:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.302+0000 7f9d58d5e700 1 -- start start 2024-03-12T03:14:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.302+0000 7f9d58d5e700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d541a6cb0 con 0x7f9d5410e6f0 2024-03-12T03:14:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.302+0000 7f9d58d5e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d541a7ed0 con 0x7f9d54102dd0 2024-03-12T03:14:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.302+0000 7f9d58d5e700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d541a9110 con 0x7f9d5410abe0 2024-03-12T03:14:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.302+0000 7f9d51d9b700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f9d5410abe0 0x7f9d541a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48404/0 (socket says 172.21.15.94:48404) 2024-03-12T03:14:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d51d9b700 1 -- 172.21.15.94:0/1695167450 learned_addr learned my addr 172.21.15.94:0/1695167450 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901732536 0 0) 0x7f9d541a9110 con 0x7f9d5410abe0 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d3c0035b0 con 0x7f9d5410abe0 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607138188 0 0) 0x7f9d541a6cb0 con 0x7f9d5410e6f0 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d541a9110 con 0x7f9d5410e6f0 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793662158 0 0) 0x7f9d541a7ed0 con 0x7f9d54102dd0 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d541a6cb0 con 0x7f9d54102dd0 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2573444523 0 0) 0x7f9d3c0035b0 con 0x7f9d5410abe0 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d541a7ed0 con 0x7f9d5410abe0 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f9d480019b0 con 0x7f9d5410abe0 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2836579759 0 0) 0x7f9d541a9110 con 0x7f9d5410e6f0 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d3c0035b0 con 0x7f9d5410e6f0 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f9d4c001c00 con 0x7f9d5410e6f0 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3495982210 0 0) 0x7f9d541a7ed0 con 0x7f9d5410abe0 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 >> v1:172.21.15.151:6789/0 conn(0x7f9d54102dd0 legacy=0x7f9d5410a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:50.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 >> v1:172.21.15.94:6789/0 conn(0x7f9d5410e6f0 legacy=0x7f9d541a5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d541aa350 con 0x7f9d5410abe0 2024-03-12T03:14:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d58d5e700 1 -- 172.21.15.94:0/1695167450 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d541a6ee0 con 0x7f9d5410abe0 2024-03-12T03:14:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d58d5e700 1 -- 172.21.15.94:0/1695167450 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d541a73a0 con 0x7f9d5410abe0 2024-03-12T03:14:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f9d48001c80 con 0x7f9d5410abe0 2024-03-12T03:14:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.303+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f9d48005cf0 con 0x7f9d5410abe0 2024-03-12T03:14:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.304+0000 7f9d58d5e700 1 -- 172.21.15.94:0/1695167450 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d34005120 con 0x7f9d5410abe0 2024-03-12T03:14:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.305+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f9d48004050 con 0x7f9d5410abe0 2024-03-12T03:14:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.305+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7f9d480891e0 con 0x7f9d5410abe0 2024-03-12T03:14:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.310+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f9d48056b60 con 0x7f9d5410abe0 2024-03-12T03:14:50.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.507+0000 7f9d58d5e700 1 -- 172.21.15.94:0/1695167450 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9d34005410 con 0x7f9d5410abe0 2024-03-12T03:14:50.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.508+0000 7f9d437fe700 1 -- 172.21.15.94:0/1695167450 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 739478307) 0x7f9d4805a5b0 con 0x7f9d5410abe0 2024-03-12T03:14:50.511 INFO:teuthology.orchestra.run.smithi094.stdout:34359738423 2024-03-12T03:14:50.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.510+0000 7f9d58d5e700 1 -- 172.21.15.94:0/1695167450 >> v1:172.21.15.94:6800/2099808905 conn(0x7f9d3c06e9c0 legacy=0x7f9d3c070e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:50.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.510+0000 7f9d58d5e700 1 -- 172.21.15.94:0/1695167450 >> v1:172.21.15.94:6790/0 conn(0x7f9d5410abe0 legacy=0x7f9d541a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:50.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.510+0000 7f9d58d5e700 1 -- 172.21.15.94:0/1695167450 shutdown_connections 2024-03-12T03:14:50.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.510+0000 7f9d58d5e700 1 -- 172.21.15.94:0/1695167450 >> 172.21.15.94:0/1695167450 conn(0x7f9d540fc580 msgr2=0x7f9d540fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:50.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.510+0000 7f9d58d5e700 1 -- 172.21.15.94:0/1695167450 shutdown_connections 2024-03-12T03:14:50.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:50.510+0000 7f9d58d5e700 1 -- 172.21.15.94:0/1695167450 wait complete. 2024-03-12T03:14:50.762 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1695167450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:14:50.762 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:50 smithi094 ceph-mon[40484]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:50.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1695167450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:14:50.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:50 smithi094 ceph-mon[29291]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:50.763 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:14:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:14:50.789 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574895 for osd.1 2024-03-12T03:14:50.789 DEBUG:teuthology.parallel:result is None 2024-03-12T03:14:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1695167450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:14:50.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:50 smithi151 ceph-mon[32923]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:51.190 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182432 got 107374182436 for osd.3 2024-03-12T03:14:51.191 DEBUG:teuthology.parallel:result is None 2024-03-12T03:14:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:51.616 INFO:teuthology.orchestra.run.smithi094.stdout:163208757274 2024-03-12T03:14:51.617 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-12T03:14:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:52 smithi094 ceph-mon[40484]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:52 smithi094 ceph-mon[29291]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:52.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:52 smithi151 ceph-mon[32923]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:53.363 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738423 for osd.0 2024-03-12T03:14:53.363 DEBUG:teuthology.parallel:result is None 2024-03-12T03:14:54.720 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:54.724 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:14:54.845 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:54 smithi094 ceph-mon[40484]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:54.845 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:54 smithi094 ceph-mon[29291]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:54 smithi151 ceph-mon[32923]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:56.644 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:56 smithi094 ceph-mon[29291]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:56.644 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:56 smithi094 ceph-mon[40484]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:56.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.644+0000 7eff6fd87700 1 -- 172.21.15.94:0/1871986214 >> v1:172.21.15.94:6789/0 conn(0x7eff6810e610 legacy=0x7eff68110ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:56.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.644+0000 7eff6fd87700 1 -- 172.21.15.94:0/1871986214 shutdown_connections 2024-03-12T03:14:56.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.644+0000 7eff6fd87700 1 -- 172.21.15.94:0/1871986214 >> 172.21.15.94:0/1871986214 conn(0x7eff680fd690 msgr2=0x7eff680ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:56.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.644+0000 7eff6fd87700 1 -- 172.21.15.94:0/1871986214 shutdown_connections 2024-03-12T03:14:56.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.644+0000 7eff6fd87700 1 -- 172.21.15.94:0/1871986214 wait complete. 2024-03-12T03:14:56.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff6fd87700 1 Processor -- start 2024-03-12T03:14:56.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff6fd87700 1 -- start start 2024-03-12T03:14:56.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff6fd87700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7eff681a8f00 con 0x7eff6810aa70 2024-03-12T03:14:56.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff6fd87700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7eff681aa120 con 0x7eff6810e610 2024-03-12T03:14:56.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff6fd87700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7eff681ab360 con 0x7eff68101de0 2024-03-12T03:14:56.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff6db23700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7eff68101de0 0x7eff68107190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:43586/0 (socket says 172.21.15.94:43586) 2024-03-12T03:14:56.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff6db23700 1 -- 172.21.15.94:0/3331308144 learned_addr learned my addr 172.21.15.94:0/3331308144 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:56.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff6d322700 1 --1- 172.21.15.94:0/3331308144 >> v1:172.21.15.94:6789/0 conn(0x7eff6810aa70 0x7eff681a3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38612/0 (socket says 172.21.15.94:38612) 2024-03-12T03:14:56.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803383348 0 0) 0x7eff681a8f00 con 0x7eff6810aa70 2024-03-12T03:14:56.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff540035b0 con 0x7eff6810aa70 2024-03-12T03:14:56.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724074707 0 0) 0x7eff681ab360 con 0x7eff68101de0 2024-03-12T03:14:56.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff681a8f00 con 0x7eff68101de0 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 48096206 0 0) 0x7eff681a8f00 con 0x7eff68101de0 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff681ab360 con 0x7eff68101de0 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7eff640013d0 con 0x7eff68101de0 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 823097076 0 0) 0x7eff540035b0 con 0x7eff6810aa70 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff681a8f00 con 0x7eff6810aa70 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.645+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7eff58001830 con 0x7eff6810aa70 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.646+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877032035 0 0) 0x7eff681aa120 con 0x7eff6810e610 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.646+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff540035b0 con 0x7eff6810e610 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.646+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 113175776 0 0) 0x7eff681ab360 con 0x7eff68101de0 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.646+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 >> v1:172.21.15.151:6789/0 conn(0x7eff6810e610 legacy=0x7eff681a75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.646+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 >> v1:172.21.15.94:6789/0 conn(0x7eff6810aa70 legacy=0x7eff681a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.646+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff681ac5a0 con 0x7eff68101de0 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.646+0000 7eff6fd87700 1 -- 172.21.15.94:0/3331308144 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff681ab590 con 0x7eff68101de0 2024-03-12T03:14:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.646+0000 7eff6fd87700 1 -- 172.21.15.94:0/3331308144 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff681aba50 con 0x7eff68101de0 2024-03-12T03:14:56.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.646+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7eff64001640 con 0x7eff68101de0 2024-03-12T03:14:56.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.646+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7eff64005d20 con 0x7eff68101de0 2024-03-12T03:14:56.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.647+0000 7eff6fd87700 1 -- 172.21.15.94:0/3331308144 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4c005120 con 0x7eff68101de0 2024-03-12T03:14:56.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.647+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7eff64003970 con 0x7eff68101de0 2024-03-12T03:14:56.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.648+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7eff64089440 con 0x7eff68101de0 2024-03-12T03:14:56.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.652+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7eff64056c80 con 0x7eff68101de0 2024-03-12T03:14:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.858+0000 7eff6fd87700 1 -- 172.21.15.94:0/3331308144 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7eff4c005410 con 0x7eff68101de0 2024-03-12T03:14:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.858+0000 7eff5effd700 1 -- 172.21.15.94:0/3331308144 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (unknown 1823589948 0 1045639440) 0x7eff6405a6d0 con 0x7eff68101de0 2024-03-12T03:14:56.861 INFO:teuthology.orchestra.run.smithi094.stdout:128849018914 2024-03-12T03:14:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.860+0000 7eff6fd87700 1 -- 172.21.15.94:0/3331308144 >> v1:172.21.15.94:6800/2099808905 conn(0x7eff5406ec30 legacy=0x7eff540710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.860+0000 7eff6fd87700 1 -- 172.21.15.94:0/3331308144 >> v1:172.21.15.94:6790/0 conn(0x7eff68101de0 legacy=0x7eff68107190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.860+0000 7eff6fd87700 1 -- 172.21.15.94:0/3331308144 shutdown_connections 2024-03-12T03:14:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.860+0000 7eff6fd87700 1 -- 172.21.15.94:0/3331308144 >> 172.21.15.94:0/3331308144 conn(0x7eff680fd690 msgr2=0x7eff68111b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.860+0000 7eff6fd87700 1 -- 172.21.15.94:0/3331308144 shutdown_connections 2024-03-12T03:14:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:56.860+0000 7eff6fd87700 1 -- 172.21.15.94:0/3331308144 wait complete. 2024-03-12T03:14:56.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:56 smithi151 ceph-mon[32923]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:14:57.936 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3331308144' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:14:57.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3331308144' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:14:57.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3331308144' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:14:57.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:14:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:14:58.210 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:14:58 smithi094 podman[75417]: 2024-03-12 03:14:58.177279311 +0000 UTC m=+0.224984260 container create edda9a59ed7ec8e9a4393da5d25fb97efca0dfc285a9664e6edc6918b90edd39 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:14:58.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-mon[40484]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-mon[29291]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:58.710 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:14:58 smithi094 podman[75417]: 2024-03-12 03:14:58.488161629 +0000 UTC m=+0.535866593 container init edda9a59ed7ec8e9a4393da5d25fb97efca0dfc285a9664e6edc6918b90edd39 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:14:58.710 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:14:58 smithi094 podman[75417]: 2024-03-12 03:14:58.490668901 +0000 UTC m=+0.538373853 container start edda9a59ed7ec8e9a4393da5d25fb97efca0dfc285a9664e6edc6918b90edd39 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:14:58.710 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[75451]: ts=2024-03-12T03:14:58.503Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-12T03:14:58.710 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[75451]: ts=2024-03-12T03:14:58.503Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-12T03:14:58.711 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[75451]: ts=2024-03-12T03:14:58.504Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.94 port=9094 2024-03-12T03:14:58.711 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[75451]: ts=2024-03-12T03:14:58.505Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-12T03:14:58.711 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[75451]: ts=2024-03-12T03:14:58.528Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:14:58.711 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[75451]: ts=2024-03-12T03:14:58.528Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:14:58.711 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[75451]: ts=2024-03-12T03:14:58.531Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-12T03:14:58.711 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[75451]: ts=2024-03-12T03:14:58.531Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-12T03:14:58.711 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:14:58 smithi094 bash[75417]: edda9a59ed7ec8e9a4393da5d25fb97efca0dfc285a9664e6edc6918b90edd39 2024-03-12T03:14:58.711 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:14:58 smithi094 systemd[1]: Started Ceph alertmanager.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:14:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:58 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:14:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:58 smithi151 ceph-mon[32923]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:14:59.210 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:14:58] ENGINE Bus STOPPING 2024-03-12T03:14:59.210 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:14:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:14:59.210 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:14:58] ENGINE Bus STOPPED 2024-03-12T03:14:59.210 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:14:58] ENGINE Bus STARTING 2024-03-12T03:14:59.210 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:14:58] ENGINE Serving on http://:::9283 2024-03-12T03:14:59.210 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:14:58 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:14:58] ENGINE Bus STARTED 2024-03-12T03:14:59.375 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018910 got 128849018914 for osd.4 2024-03-12T03:14:59.375 DEBUG:teuthology.parallel:result is None 2024-03-12T03:14:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.619+0000 7f8b0a30e700 1 -- 172.21.15.94:0/1204371734 >> v1:172.21.15.94:6790/0 conn(0x7f8b0410aa40 legacy=0x7f8b0410ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.620+0000 7f8b0a30e700 1 -- 172.21.15.94:0/1204371734 shutdown_connections 2024-03-12T03:14:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.620+0000 7f8b0a30e700 1 -- 172.21.15.94:0/1204371734 >> 172.21.15.94:0/1204371734 conn(0x7f8b04075dd0 msgr2=0x7f8b040761e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.620+0000 7f8b0a30e700 1 -- 172.21.15.94:0/1204371734 shutdown_connections 2024-03-12T03:14:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.620+0000 7f8b0a30e700 1 -- 172.21.15.94:0/1204371734 wait complete. 2024-03-12T03:14:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.620+0000 7f8b0a30e700 1 Processor -- start 2024-03-12T03:14:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.620+0000 7f8b0a30e700 1 -- start start 2024-03-12T03:14:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b0a30e700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8b041073c0 con 0x7f8b0410aa40 2024-03-12T03:14:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b0a30e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8b041075d0 con 0x7f8b0410e640 2024-03-12T03:14:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b0a30e700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8b041a9260 con 0x7f8b04107cd0 2024-03-12T03:14:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b037fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8b0410aa40 0x7f8b041049c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38628/0 (socket says 172.21.15.94:38628) 2024-03-12T03:14:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b03fff700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f8b04107cd0 0x7f8b041042b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:43592/0 (socket says 172.21.15.94:43592) 2024-03-12T03:14:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b037fe700 1 -- 172.21.15.94:0/3340641958 learned_addr learned my addr 172.21.15.94:0/3340641958 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:14:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599734819 0 0) 0x7f8b041073c0 con 0x7f8b0410aa40 2024-03-12T03:14:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aec0035b0 con 0x7f8b0410aa40 2024-03-12T03:14:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781608503 0 0) 0x7f8b041075d0 con 0x7f8b0410e640 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b041073c0 con 0x7f8b0410e640 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165690421 0 0) 0x7f8b041a9260 con 0x7f8b04107cd0 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b041075d0 con 0x7f8b04107cd0 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4087408315 0 0) 0x7f8aec0035b0 con 0x7f8b0410aa40 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b041a9260 con 0x7f8b0410aa40 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.621+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8af8001a20 con 0x7f8b0410aa40 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1260042376 0 0) 0x7f8b041073c0 con 0x7f8b0410e640 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8aec0035b0 con 0x7f8b0410e640 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1763514457 0 0) 0x7f8b041075d0 con 0x7f8b04107cd0 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b041073c0 con 0x7f8b04107cd0 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8afc001b00 con 0x7f8b0410e640 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8af40017b0 con 0x7f8b04107cd0 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2801564977 0 0) 0x7f8b041a9260 con 0x7f8b0410aa40 2024-03-12T03:14:59.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 >> v1:172.21.15.94:6790/0 conn(0x7f8b04107cd0 legacy=0x7f8b041042b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 >> v1:172.21.15.151:6789/0 conn(0x7f8b0410e640 legacy=0x7f8b041a5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b041aa480 con 0x7f8b0410aa40 2024-03-12T03:14:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b0a30e700 1 -- 172.21.15.94:0/3340641958 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b041a9490 con 0x7f8b0410aa40 2024-03-12T03:14:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b0a30e700 1 -- 172.21.15.94:0/3340641958 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b041a9950 con 0x7f8b0410aa40 2024-03-12T03:14:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f8af80011c0 con 0x7f8b0410aa40 2024-03-12T03:14:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.622+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8af8004de0 con 0x7f8b0410aa40 2024-03-12T03:14:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.623+0000 7f8b0a30e700 1 -- 172.21.15.94:0/3340641958 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af0005120 con 0x7f8b0410aa40 2024-03-12T03:14:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.623+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f8af8005020 con 0x7f8b0410aa40 2024-03-12T03:14:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.624+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7f8af8089210 con 0x7f8b0410aa40 2024-03-12T03:14:59.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.629+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f8af8056a90 con 0x7f8b0410aa40 2024-03-12T03:14:59.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.827+0000 7f8b0a30e700 1 -- 172.21.15.94:0/3340641958 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f8af0005410 con 0x7f8b0410aa40 2024-03-12T03:14:59.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.827+0000 7f8b017fa700 1 -- 172.21.15.94:0/3340641958 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 3109443431) 0x7f8af8004dc0 con 0x7f8b0410aa40 2024-03-12T03:14:59.830 INFO:teuthology.orchestra.run.smithi094.stdout:77309411374 2024-03-12T03:14:59.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.829+0000 7f8b0a30e700 1 -- 172.21.15.94:0/3340641958 >> v1:172.21.15.94:6800/2099808905 conn(0x7f8aec06ec30 legacy=0x7f8aec0710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:59.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.829+0000 7f8b0a30e700 1 -- 172.21.15.94:0/3340641958 >> v1:172.21.15.94:6789/0 conn(0x7f8b0410aa40 legacy=0x7f8b041049c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:14:59.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.830+0000 7f8b0a30e700 1 -- 172.21.15.94:0/3340641958 shutdown_connections 2024-03-12T03:14:59.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.830+0000 7f8b0a30e700 1 -- 172.21.15.94:0/3340641958 >> 172.21.15.94:0/3340641958 conn(0x7f8b04075dd0 msgr2=0x7f8b04110aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:14:59.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.830+0000 7f8b0a30e700 1 -- 172.21.15.94:0/3340641958 shutdown_connections 2024-03-12T03:14:59.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:14:59.830+0000 7f8b0a30e700 1 -- 172.21.15.94:0/3340641958 wait complete. 2024-03-12T03:14:59.877 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.877 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.877 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.877 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.877 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[40484]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T03:14:59.877 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.877 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.877 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:14:59.878 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:14:59.878 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.878 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[40484]: Deploying daemon grafana.a on smithi151 2024-03-12T03:14:59.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[29291]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T03:14:59.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:14:59.879 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:14:59.879 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.879 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:14:59 smithi094 ceph-mon[29291]: Deploying daemon grafana.a on smithi151 2024-03-12T03:14:59.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:59 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:59 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:59 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:59 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:59 smithi151 ceph-mon[32923]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T03:14:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:59 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:59 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:59 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:14:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:59 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:14:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:59 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:14:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:14:59 smithi151 ceph-mon[32923]: Deploying daemon grafana.a on smithi151 2024-03-12T03:15:00.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.335+0000 7f9d45a84700 1 -- 172.21.15.94:0/1317351473 >> v1:172.21.15.94:6789/0 conn(0x7f9d4010d5e0 legacy=0x7f9d4010fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:00.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.335+0000 7f9d45a84700 1 -- 172.21.15.94:0/1317351473 shutdown_connections 2024-03-12T03:15:00.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.335+0000 7f9d45a84700 1 -- 172.21.15.94:0/1317351473 >> 172.21.15.94:0/1317351473 conn(0x7f9d400fc640 msgr2=0x7f9d400fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:00.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.335+0000 7f9d45a84700 1 -- 172.21.15.94:0/1317351473 shutdown_connections 2024-03-12T03:15:00.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.336+0000 7f9d45a84700 1 -- 172.21.15.94:0/1317351473 wait complete. 2024-03-12T03:15:00.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.336+0000 7f9d45a84700 1 Processor -- start 2024-03-12T03:15:00.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.336+0000 7f9d45a84700 1 -- start start 2024-03-12T03:15:00.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.336+0000 7f9d45a84700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d401a6c50 con 0x7f9d401099b0 2024-03-12T03:15:00.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.336+0000 7f9d45a84700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d401a7e70 con 0x7f9d40106c00 2024-03-12T03:15:00.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.336+0000 7f9d45a84700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d401a90b0 con 0x7f9d4010d5e0 2024-03-12T03:15:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3effd700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9d401099b0 0x7f9d401a1a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38650/0 (socket says 172.21.15.94:38650) 2024-03-12T03:15:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3effd700 1 -- 172.21.15.94:0/3482772035 learned_addr learned my addr 172.21.15.94:0/3482772035 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:15:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817701040 0 0) 0x7f9d401a6c50 con 0x7f9d401099b0 2024-03-12T03:15:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d280035b0 con 0x7f9d401099b0 2024-03-12T03:15:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448527537 0 0) 0x7f9d401a90b0 con 0x7f9d4010d5e0 2024-03-12T03:15:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d401a6c50 con 0x7f9d4010d5e0 2024-03-12T03:15:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317596121 0 0) 0x7f9d401a7e70 con 0x7f9d40106c00 2024-03-12T03:15:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d401a90b0 con 0x7f9d40106c00 2024-03-12T03:15:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3592361967 0 0) 0x7f9d280035b0 con 0x7f9d401099b0 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d401a7e70 con 0x7f9d401099b0 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2036610492 0 0) 0x7f9d401a6c50 con 0x7f9d4010d5e0 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d280035b0 con 0x7f9d4010d5e0 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f9d34001910 con 0x7f9d401099b0 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f9d38004740 con 0x7f9d4010d5e0 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2245710219 0 0) 0x7f9d401a90b0 con 0x7f9d40106c00 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d401a6c50 con 0x7f9d40106c00 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f9d300017f0 con 0x7f9d40106c00 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1811694211 0 0) 0x7f9d401a7e70 con 0x7f9d401099b0 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 >> v1:172.21.15.94:6790/0 conn(0x7f9d4010d5e0 legacy=0x7f9d401a5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 >> v1:172.21.15.151:6789/0 conn(0x7f9d40106c00 legacy=0x7f9d40105f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d401aa2f0 con 0x7f9d401099b0 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d45a84700 1 -- 172.21.15.94:0/3482772035 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d401a80a0 con 0x7f9d401099b0 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.337+0000 7f9d45a84700 1 -- 172.21.15.94:0/3482772035 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d401a8560 con 0x7f9d401099b0 2024-03-12T03:15:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.338+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f9d34001ce0 con 0x7f9d401099b0 2024-03-12T03:15:00.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.338+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f9d34005d50 con 0x7f9d401099b0 2024-03-12T03:15:00.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.338+0000 7f9d45a84700 1 -- 172.21.15.94:0/3482772035 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d2c005120 con 0x7f9d401099b0 2024-03-12T03:15:00.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.339+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f9d34004070 con 0x7f9d401099b0 2024-03-12T03:15:00.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.339+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7f9d340891e0 con 0x7f9d401099b0 2024-03-12T03:15:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.345+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f9d34056a60 con 0x7f9d401099b0 2024-03-12T03:15:00.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.542+0000 7f9d45a84700 1 -- 172.21.15.94:0/3482772035 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9d2c005410 con 0x7f9d401099b0 2024-03-12T03:15:00.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.542+0000 7f9d3cff9700 1 -- 172.21.15.94:0/3482772035 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (unknown 2131975755 0 4260725640) 0x7f9d3405a4b0 con 0x7f9d401099b0 2024-03-12T03:15:00.545 INFO:teuthology.orchestra.run.smithi094.stdout:163208757278 2024-03-12T03:15:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.544+0000 7f9d45a84700 1 -- 172.21.15.94:0/3482772035 >> v1:172.21.15.94:6800/2099808905 conn(0x7f9d2806eca0 legacy=0x7f9d28071160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.544+0000 7f9d45a84700 1 -- 172.21.15.94:0/3482772035 >> v1:172.21.15.94:6789/0 conn(0x7f9d401099b0 legacy=0x7f9d401a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.544+0000 7f9d45a84700 1 -- 172.21.15.94:0/3482772035 shutdown_connections 2024-03-12T03:15:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.544+0000 7f9d45a84700 1 -- 172.21.15.94:0/3482772035 >> 172.21.15.94:0/3482772035 conn(0x7f9d400fc640 msgr2=0x7f9d400fca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.544+0000 7f9d45a84700 1 -- 172.21.15.94:0/3482772035 shutdown_connections 2024-03-12T03:15:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:00.544+0000 7f9d45a84700 1 -- 172.21.15.94:0/3482772035 wait complete. 2024-03-12T03:15:00.758 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411371 got 77309411374 for osd.2 2024-03-12T03:15:00.758 DEBUG:teuthology.parallel:result is None 2024-03-12T03:15:00.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:15:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:15:00.960 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[75451]: ts=2024-03-12T03:15:00.505Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000524405s 2024-03-12T03:15:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3340641958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:15:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:00 smithi094 ceph-mon[29291]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3482772035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:15:00.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3340641958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:15:00.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:00 smithi094 ceph-mon[40484]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:00.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3482772035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:15:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3340641958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:15:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:00 smithi151 ceph-mon[32923]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3482772035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:15:01.441 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757274 got 163208757278 for osd.5 2024-03-12T03:15:01.442 DEBUG:teuthology.parallel:result is None 2024-03-12T03:15:01.442 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-12T03:15:01.442 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:15:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:15:02.955 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:02 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:02.955 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:02 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:15:02.955 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:02 smithi094 ceph-mon[29291]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:02.955 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:02 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:02.955 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:02 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:15:02.955 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:02 smithi094 ceph-mon[40484]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:02 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:02 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:15:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:02 smithi151 ceph-mon[32923]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:03.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.483+0000 7fee2e8b4700 1 -- 172.21.15.94:0/3642132157 >> v1:172.21.15.94:6789/0 conn(0x7fee28109920 legacy=0x7fee2810bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:03.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.483+0000 7fee2e8b4700 1 -- 172.21.15.94:0/3642132157 shutdown_connections 2024-03-12T03:15:03.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.483+0000 7fee2e8b4700 1 -- 172.21.15.94:0/3642132157 >> 172.21.15.94:0/3642132157 conn(0x7fee280fc580 msgr2=0x7fee280fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:03.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.483+0000 7fee2e8b4700 1 -- 172.21.15.94:0/3642132157 shutdown_connections 2024-03-12T03:15:03.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.483+0000 7fee2e8b4700 1 -- 172.21.15.94:0/3642132157 wait complete. 2024-03-12T03:15:03.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.483+0000 7fee2e8b4700 1 Processor -- start 2024-03-12T03:15:03.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.483+0000 7fee2e8b4700 1 -- start start 2024-03-12T03:15:03.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee2e8b4700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fee281a6c20 con 0x7fee28106a70 2024-03-12T03:15:03.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee2e8b4700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fee281a7e40 con 0x7fee28109920 2024-03-12T03:15:03.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee2e8b4700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fee281a9080 con 0x7fee2810d4f0 2024-03-12T03:15:03.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee27fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fee28106a70 0x7fee2810cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38656/0 (socket says 172.21.15.94:38656) 2024-03-12T03:15:03.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee27fff700 1 -- 172.21.15.94:0/1503368215 learned_addr learned my addr 172.21.15.94:0/1503368215 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:15:03.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547311310 0 0) 0x7fee281a6c20 con 0x7fee28106a70 2024-03-12T03:15:03.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee100035b0 con 0x7fee28106a70 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800817161 0 0) 0x7fee281a9080 con 0x7fee2810d4f0 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee281a6c20 con 0x7fee2810d4f0 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145056254 0 0) 0x7fee281a7e40 con 0x7fee28109920 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee281a9080 con 0x7fee28109920 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2274751452 0 0) 0x7fee100035b0 con 0x7fee28106a70 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee281a7e40 con 0x7fee28106a70 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1632906781 0 0) 0x7fee281a6c20 con 0x7fee2810d4f0 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee100035b0 con 0x7fee2810d4f0 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.484+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fee18001a10 con 0x7fee28106a70 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.485+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fee20001bb0 con 0x7fee2810d4f0 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.485+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 361750923 0 0) 0x7fee281a7e40 con 0x7fee28106a70 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.485+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 >> v1:172.21.15.94:6790/0 conn(0x7fee2810d4f0 legacy=0x7fee281a52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.485+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 >> v1:172.21.15.151:6789/0 conn(0x7fee28109920 legacy=0x7fee281a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:03.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.485+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee281aa2c0 con 0x7fee28106a70 2024-03-12T03:15:03.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.485+0000 7fee2e8b4700 1 -- 172.21.15.94:0/1503368215 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee281a92b0 con 0x7fee28106a70 2024-03-12T03:15:03.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.485+0000 7fee2e8b4700 1 -- 172.21.15.94:0/1503368215 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee281a9770 con 0x7fee28106a70 2024-03-12T03:15:03.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.485+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fee18001d30 con 0x7fee28106a70 2024-03-12T03:15:03.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.485+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fee18005d10 con 0x7fee28106a70 2024-03-12T03:15:03.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.486+0000 7fee2e8b4700 1 -- 172.21.15.94:0/1503368215 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee14005120 con 0x7fee28106a70 2024-03-12T03:15:03.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.486+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7fee18003930 con 0x7fee28106a70 2024-03-12T03:15:03.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.487+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7fee18089150 con 0x7fee28106a70 2024-03-12T03:15:03.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.491+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fee18056990 con 0x7fee28106a70 2024-03-12T03:15:03.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.683+0000 7fee2e8b4700 1 -- 172.21.15.94:0/1503368215 --> v1:172.21.15.94:6800/2099808905 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee14000b90 con 0x7fee1006e9c0 2024-03-12T03:15:03.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.694+0000 7fee257fa700 1 -- 172.21.15.94:0/1503368215 <== mgr.34490 v1:172.21.15.94:6800/2099808905 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346507 (unknown 2965378022 0 2194321947) 0x7fee14000b90 con 0x7fee1006e9c0 2024-03-12T03:15:03.702 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:15:03.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.698+0000 7fee2e8b4700 1 -- 172.21.15.94:0/1503368215 >> v1:172.21.15.94:6800/2099808905 conn(0x7fee1006e9c0 legacy=0x7fee10070e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:03.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.698+0000 7fee2e8b4700 1 -- 172.21.15.94:0/1503368215 >> v1:172.21.15.94:6789/0 conn(0x7fee28106a70 legacy=0x7fee2810cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:03.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.699+0000 7fee2e8b4700 1 -- 172.21.15.94:0/1503368215 shutdown_connections 2024-03-12T03:15:03.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.699+0000 7fee2e8b4700 1 -- 172.21.15.94:0/1503368215 >> 172.21.15.94:0/1503368215 conn(0x7fee280fc580 msgr2=0x7fee28100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:03.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.699+0000 7fee2e8b4700 1 -- 172.21.15.94:0/1503368215 shutdown_connections 2024-03-12T03:15:03.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:03.699+0000 7fee2e8b4700 1 -- 172.21.15.94:0/1503368215 wait complete. 2024-03-12T03:15:03.705 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-12T03:15:04.490 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-12T03:15:02.528957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1299,"num_read_kb":1113,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":231864,"kb_used_data":7388,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749729352,"statfs":{"total":767960285184,"available":767722856448,"internally_reserved":0,"allocated":7565312,"data_stored":4028136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":229756465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002841"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510147+0000","last_change":"2024-03-12T03:13:44.944304+0000","last_active":"2024-03-12T03:14:16.510147+0000","last_peered":"2024-03-12T03:14:16.510147+0000","last_clean":"2024-03-12T03:14:16.510147+0000","last_became_active":"2024-03-12T03:13:44.943975+0000","last_became_peered":"2024-03-12T03:13:44.943975+0000","last_unstale":"2024-03-12T03:14:16.510147+0000","last_undegraded":"2024-03-12T03:14:16.510147+0000","last_fullsized":"2024-03-12T03:14:16.510147+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:05:41.838808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.508913+0000","last_change":"2024-03-12T03:13:42.940899+0000","last_active":"2024-03-12T03:14:16.508913+0000","last_peered":"2024-03-12T03:14:16.508913+0000","last_clean":"2024-03-12T03:14:16.508913+0000","last_became_active":"2024-03-12T03:13:42.940414+0000","last_became_peered":"2024-03-12T03:13:42.940414+0000","last_unstale":"2024-03-12T03:14:16.508913+0000","last_undegraded":"2024-03-12T03:14:16.508913+0000","last_fullsized":"2024-03-12T03:14:16.508913+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:52:38.732745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999795+0000","last_change":"2024-03-12T03:13:38.913213+0000","last_active":"2024-03-12T03:14:16.999795+0000","last_peered":"2024-03-12T03:14:16.999795+0000","last_clean":"2024-03-12T03:14:16.999795+0000","last_became_active":"2024-03-12T03:13:38.912803+0000","last_became_peered":"2024-03-12T03:13:38.912803+0000","last_unstale":"2024-03-12T03:14:16.999795+0000","last_undegraded":"2024-03-12T03:14:16.999795+0000","last_fullsized":"2024-03-12T03:14:16.999795+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:49:25.201096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'15","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509886+0000","last_change":"2024-03-12T03:13:40.924525+0000","last_active":"2024-03-12T03:14:16.509886+0000","last_peered":"2024-03-12T03:14:16.509886+0000","last_clean":"2024-03-12T03:14:16.509886+0000","last_became_active":"2024-03-12T03:13:40.923702+0000","last_became_peered":"2024-03-12T03:13:40.923702+0000","last_unstale":"2024-03-12T03:14:16.509886+0000","last_undegraded":"2024-03-12T03:14:16.509886+0000","last_fullsized":"2024-03-12T03:14:16.509886+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:45:47.871678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510317+0000","last_change":"2024-03-12T03:13:44.999070+0000","last_active":"2024-03-12T03:14:16.510317+0000","last_peered":"2024-03-12T03:14:16.510317+0000","last_clean":"2024-03-12T03:14:16.510317+0000","last_became_active":"2024-03-12T03:13:44.998667+0000","last_became_peered":"2024-03-12T03:13:44.998667+0000","last_unstale":"2024-03-12T03:14:16.510317+0000","last_undegraded":"2024-03-12T03:14:16.510317+0000","last_fullsized":"2024-03-12T03:14:16.510317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:48:01.866371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509889+0000","last_change":"2024-03-12T03:13:42.937335+0000","last_active":"2024-03-12T03:14:16.509889+0000","last_peered":"2024-03-12T03:14:16.509889+0000","last_clean":"2024-03-12T03:14:16.509889+0000","last_became_active":"2024-03-12T03:13:42.934402+0000","last_became_peered":"2024-03-12T03:13:42.934402+0000","last_unstale":"2024-03-12T03:14:16.509889+0000","last_undegraded":"2024-03-12T03:14:16.509889+0000","last_fullsized":"2024-03-12T03:14:16.509889+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:21:24.114718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.511864+0000","last_change":"2024-03-12T03:13:38.924780+0000","last_active":"2024-03-12T03:14:16.511864+0000","last_peered":"2024-03-12T03:14:16.511864+0000","last_clean":"2024-03-12T03:14:16.511864+0000","last_became_active":"2024-03-12T03:13:38.924533+0000","last_became_peered":"2024-03-12T03:13:38.924533+0000","last_unstale":"2024-03-12T03:14:16.511864+0000","last_undegraded":"2024-03-12T03:14:16.511864+0000","last_fullsized":"2024-03-12T03:14:16.511864+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:30.316175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'39","reported_seq":102,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.513216+0000","last_change":"2024-03-12T03:13:40.928759+0000","last_active":"2024-03-12T03:14:16.513216+0000","last_peered":"2024-03-12T03:14:16.513216+0000","last_clean":"2024-03-12T03:14:16.513216+0000","last_became_active":"2024-03-12T03:13:40.928473+0000","last_became_peered":"2024-03-12T03:13:40.928473+0000","last_unstale":"2024-03-12T03:14:16.513216+0000","last_undegraded":"2024-03-12T03:14:16.513216+0000","last_fullsized":"2024-03-12T03:14:16.513216+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:00:02.208063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.998591+0000","last_change":"2024-03-12T03:13:44.945990+0000","last_active":"2024-03-12T03:14:16.998591+0000","last_peered":"2024-03-12T03:14:16.998591+0000","last_clean":"2024-03-12T03:14:16.998591+0000","last_became_active":"2024-03-12T03:13:44.945653+0000","last_became_peered":"2024-03-12T03:13:44.945653+0000","last_unstale":"2024-03-12T03:14:16.998591+0000","last_undegraded":"2024-03-12T03:14:16.998591+0000","last_fullsized":"2024-03-12T03:14:16.998591+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:50:14.933362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615558+0000","last_change":"2024-03-12T03:13:42.941836+0000","last_active":"2024-03-12T03:14:16.615558+0000","last_peered":"2024-03-12T03:14:16.615558+0000","last_clean":"2024-03-12T03:14:16.615558+0000","last_became_active":"2024-03-12T03:13:42.940973+0000","last_became_peered":"2024-03-12T03:13:42.940973+0000","last_unstale":"2024-03-12T03:14:16.615558+0000","last_undegraded":"2024-03-12T03:14:16.615558+0000","last_fullsized":"2024-03-12T03:14:16.615558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:39:55.057516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615737+0000","last_change":"2024-03-12T03:13:38.919541+0000","last_active":"2024-03-12T03:14:16.615737+0000","last_peered":"2024-03-12T03:14:16.615737+0000","last_clean":"2024-03-12T03:14:16.615737+0000","last_became_active":"2024-03-12T03:13:38.919242+0000","last_became_peered":"2024-03-12T03:13:38.919242+0000","last_unstale":"2024-03-12T03:14:16.615737+0000","last_undegraded":"2024-03-12T03:14:16.615737+0000","last_fullsized":"2024-03-12T03:14:16.615737+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:50:27.245276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'21","reported_seq":69,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510393+0000","last_change":"2024-03-12T03:13:40.926045+0000","last_active":"2024-03-12T03:14:16.510393+0000","last_peered":"2024-03-12T03:14:16.510393+0000","last_clean":"2024-03-12T03:14:16.510393+0000","last_became_active":"2024-03-12T03:13:40.925733+0000","last_became_peered":"2024-03-12T03:13:40.925733+0000","last_unstale":"2024-03-12T03:14:16.510393+0000","last_undegraded":"2024-03-12T03:14:16.510393+0000","last_fullsized":"2024-03-12T03:14:16.510393+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:18:36.134706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615317+0000","last_change":"2024-03-12T03:13:44.940804+0000","last_active":"2024-03-12T03:14:16.615317+0000","last_peered":"2024-03-12T03:14:16.615317+0000","last_clean":"2024-03-12T03:14:16.615317+0000","last_became_active":"2024-03-12T03:13:44.939677+0000","last_became_peered":"2024-03-12T03:13:44.939677+0000","last_unstale":"2024-03-12T03:14:16.615317+0000","last_undegraded":"2024-03-12T03:14:16.615317+0000","last_fullsized":"2024-03-12T03:14:16.615317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:06:06.856376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.626990+0000","last_change":"2024-03-12T03:13:42.941547+0000","last_active":"2024-03-12T03:14:16.626990+0000","last_peered":"2024-03-12T03:14:16.626990+0000","last_clean":"2024-03-12T03:14:16.626990+0000","last_became_active":"2024-03-12T03:13:42.941131+0000","last_became_peered":"2024-03-12T03:13:42.941131+0000","last_unstale":"2024-03-12T03:14:16.626990+0000","last_undegraded":"2024-03-12T03:14:16.626990+0000","last_fullsized":"2024-03-12T03:14:16.626990+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:16:33.264911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615610+0000","last_change":"2024-03-12T03:13:38.921507+0000","last_active":"2024-03-12T03:14:16.615610+0000","last_peered":"2024-03-12T03:14:16.615610+0000","last_clean":"2024-03-12T03:14:16.615610+0000","last_became_active":"2024-03-12T03:13:38.921229+0000","last_became_peered":"2024-03-12T03:13:38.921229+0000","last_unstale":"2024-03-12T03:14:16.615610+0000","last_undegraded":"2024-03-12T03:14:16.615610+0000","last_fullsized":"2024-03-12T03:14:16.615610+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:05:06.491743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'11","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510287+0000","last_change":"2024-03-12T03:13:40.922981+0000","last_active":"2024-03-12T03:14:16.510287+0000","last_peered":"2024-03-12T03:14:16.510287+0000","last_clean":"2024-03-12T03:14:16.510287+0000","last_became_active":"2024-03-12T03:13:40.922478+0000","last_became_peered":"2024-03-12T03:13:40.922478+0000","last_unstale":"2024-03-12T03:14:16.510287+0000","last_undegraded":"2024-03-12T03:14:16.510287+0000","last_fullsized":"2024-03-12T03:14:16.510287+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:08.463482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510795+0000","last_change":"2024-03-12T03:13:44.997335+0000","last_active":"2024-03-12T03:14:16.510795+0000","last_peered":"2024-03-12T03:14:16.510795+0000","last_clean":"2024-03-12T03:14:16.510795+0000","last_became_active":"2024-03-12T03:13:44.996990+0000","last_became_peered":"2024-03-12T03:13:44.996990+0000","last_unstale":"2024-03-12T03:14:16.510795+0000","last_undegraded":"2024-03-12T03:14:16.510795+0000","last_fullsized":"2024-03-12T03:14:16.510795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:45:35.639293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510785+0000","last_change":"2024-03-12T03:13:42.940699+0000","last_active":"2024-03-12T03:14:16.510785+0000","last_peered":"2024-03-12T03:14:16.510785+0000","last_clean":"2024-03-12T03:14:16.510785+0000","last_became_active":"2024-03-12T03:13:42.940381+0000","last_became_peered":"2024-03-12T03:13:42.940381+0000","last_unstale":"2024-03-12T03:14:16.510785+0000","last_undegraded":"2024-03-12T03:14:16.510785+0000","last_fullsized":"2024-03-12T03:14:16.510785+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:42:11.488043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000775+0000","last_change":"2024-03-12T03:13:38.925929+0000","last_active":"2024-03-12T03:14:17.000775+0000","last_peered":"2024-03-12T03:14:17.000775+0000","last_clean":"2024-03-12T03:14:17.000775+0000","last_became_active":"2024-03-12T03:13:38.925023+0000","last_became_peered":"2024-03-12T03:13:38.925023+0000","last_unstale":"2024-03-12T03:14:17.000775+0000","last_undegraded":"2024-03-12T03:14:17.000775+0000","last_fullsized":"2024-03-12T03:14:17.000775+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:23.058114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.624291+0000","last_change":"2024-03-12T03:13:40.925810+0000","last_active":"2024-03-12T03:14:16.624291+0000","last_peered":"2024-03-12T03:14:16.624291+0000","last_clean":"2024-03-12T03:14:16.624291+0000","last_became_active":"2024-03-12T03:13:40.925512+0000","last_became_peered":"2024-03-12T03:13:40.925512+0000","last_unstale":"2024-03-12T03:14:16.624291+0000","last_undegraded":"2024-03-12T03:14:16.624291+0000","last_fullsized":"2024-03-12T03:14:16.624291+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:31:00.134618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.512434+0000","last_change":"2024-03-12T03:13:44.997361+0000","last_active":"2024-03-12T03:14:16.512434+0000","last_peered":"2024-03-12T03:14:16.512434+0000","last_clean":"2024-03-12T03:14:16.512434+0000","last_became_active":"2024-03-12T03:13:44.997016+0000","last_became_peered":"2024-03-12T03:13:44.997016+0000","last_unstale":"2024-03-12T03:14:16.512434+0000","last_undegraded":"2024-03-12T03:14:16.512434+0000","last_fullsized":"2024-03-12T03:14:16.512434+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:40:03.641863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510815+0000","last_change":"2024-03-12T03:13:42.940153+0000","last_active":"2024-03-12T03:14:16.510815+0000","last_peered":"2024-03-12T03:14:16.510815+0000","last_clean":"2024-03-12T03:14:16.510815+0000","last_became_active":"2024-03-12T03:13:42.939411+0000","last_became_peered":"2024-03-12T03:13:42.939411+0000","last_unstale":"2024-03-12T03:14:16.510815+0000","last_undegraded":"2024-03-12T03:14:16.510815+0000","last_fullsized":"2024-03-12T03:14:16.510815+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:55:41.812233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510532+0000","last_change":"2024-03-12T03:13:38.921057+0000","last_active":"2024-03-12T03:14:16.510532+0000","last_peered":"2024-03-12T03:14:16.510532+0000","last_clean":"2024-03-12T03:14:16.510532+0000","last_became_active":"2024-03-12T03:13:38.920757+0000","last_became_peered":"2024-03-12T03:13:38.920757+0000","last_unstale":"2024-03-12T03:14:16.510532+0000","last_undegraded":"2024-03-12T03:14:16.510532+0000","last_fullsized":"2024-03-12T03:14:16.510532+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:25:50.682120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'18","reported_seq":84,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.513044+0000","last_change":"2024-03-12T03:14:04.012274+0000","last_active":"2024-03-12T03:14:16.513044+0000","last_peered":"2024-03-12T03:14:16.513044+0000","last_clean":"2024-03-12T03:14:16.513044+0000","last_became_active":"2024-03-12T03:14:04.005636+0000","last_became_peered":"2024-03-12T03:14:04.005636+0000","last_unstale":"2024-03-12T03:14:16.513044+0000","last_undegraded":"2024-03-12T03:14:16.513044+0000","last_fullsized":"2024-03-12T03:14:16.513044+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:16:16.943541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627359+0000","last_change":"2024-03-12T03:13:44.949139+0000","last_active":"2024-03-12T03:14:16.627359+0000","last_peered":"2024-03-12T03:14:16.627359+0000","last_clean":"2024-03-12T03:14:16.627359+0000","last_became_active":"2024-03-12T03:13:44.948626+0000","last_became_peered":"2024-03-12T03:13:44.948626+0000","last_unstale":"2024-03-12T03:14:16.627359+0000","last_undegraded":"2024-03-12T03:14:16.627359+0000","last_fullsized":"2024-03-12T03:14:16.627359+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:35:45.981328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.982068+0000","last_change":"2024-03-12T03:13:42.944817+0000","last_active":"2024-03-12T03:14:53.982068+0000","last_peered":"2024-03-12T03:14:53.982068+0000","last_clean":"2024-03-12T03:14:53.982068+0000","last_became_active":"2024-03-12T03:13:42.944163+0000","last_became_peered":"2024-03-12T03:13:42.944163+0000","last_unstale":"2024-03-12T03:14:53.982068+0000","last_undegraded":"2024-03-12T03:14:53.982068+0000","last_fullsized":"2024-03-12T03:14:53.982068+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:07:51.947025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509005+0000","last_change":"2024-03-12T03:13:38.921827+0000","last_active":"2024-03-12T03:14:16.509005+0000","last_peered":"2024-03-12T03:14:16.509005+0000","last_clean":"2024-03-12T03:14:16.509005+0000","last_became_active":"2024-03-12T03:13:38.921535+0000","last_became_peered":"2024-03-12T03:13:38.921535+0000","last_unstale":"2024-03-12T03:14:16.509005+0000","last_undegraded":"2024-03-12T03:14:16.509005+0000","last_fullsized":"2024-03-12T03:14:16.509005+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:13:46.740854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'16","reported_seq":56,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000224+0000","last_change":"2024-03-12T03:13:41.098610+0000","last_active":"2024-03-12T03:14:17.000224+0000","last_peered":"2024-03-12T03:14:17.000224+0000","last_clean":"2024-03-12T03:14:17.000224+0000","last_became_active":"2024-03-12T03:13:41.098331+0000","last_became_peered":"2024-03-12T03:13:41.098331+0000","last_unstale":"2024-03-12T03:14:17.000224+0000","last_undegraded":"2024-03-12T03:14:17.000224+0000","last_fullsized":"2024-03-12T03:14:17.000224+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:48:16.277359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.508721+0000","last_change":"2024-03-12T03:13:44.942886+0000","last_active":"2024-03-12T03:14:16.508721+0000","last_peered":"2024-03-12T03:14:16.508721+0000","last_clean":"2024-03-12T03:14:16.508721+0000","last_became_active":"2024-03-12T03:13:44.942558+0000","last_became_peered":"2024-03-12T03:13:44.942558+0000","last_unstale":"2024-03-12T03:14:16.508721+0000","last_undegraded":"2024-03-12T03:14:16.508721+0000","last_fullsized":"2024-03-12T03:14:16.508721+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:34:51.142099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000370+0000","last_change":"2024-03-12T03:13:42.936686+0000","last_active":"2024-03-12T03:14:17.000370+0000","last_peered":"2024-03-12T03:14:17.000370+0000","last_clean":"2024-03-12T03:14:17.000370+0000","last_became_active":"2024-03-12T03:13:42.935514+0000","last_became_peered":"2024-03-12T03:13:42.935514+0000","last_unstale":"2024-03-12T03:14:17.000370+0000","last_undegraded":"2024-03-12T03:14:17.000370+0000","last_fullsized":"2024-03-12T03:14:17.000370+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:15:03.926327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509528+0000","last_change":"2024-03-12T03:13:38.927276+0000","last_active":"2024-03-12T03:14:16.509528+0000","last_peered":"2024-03-12T03:14:16.509528+0000","last_clean":"2024-03-12T03:14:16.509528+0000","last_became_active":"2024-03-12T03:13:38.926916+0000","last_became_peered":"2024-03-12T03:13:38.926916+0000","last_unstale":"2024-03-12T03:14:16.509528+0000","last_undegraded":"2024-03-12T03:14:16.509528+0000","last_fullsized":"2024-03-12T03:14:16.509528+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:19:14.943793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'17","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999850+0000","last_change":"2024-03-12T03:13:40.929310+0000","last_active":"2024-03-12T03:14:16.999850+0000","last_peered":"2024-03-12T03:14:16.999850+0000","last_clean":"2024-03-12T03:14:16.999850+0000","last_became_active":"2024-03-12T03:13:40.928990+0000","last_became_peered":"2024-03-12T03:13:40.928990+0000","last_unstale":"2024-03-12T03:14:16.999850+0000","last_undegraded":"2024-03-12T03:14:16.999850+0000","last_fullsized":"2024-03-12T03:14:16.999850+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:33:36.214911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.626996+0000","last_change":"2024-03-12T03:13:44.946379+0000","last_active":"2024-03-12T03:14:16.626996+0000","last_peered":"2024-03-12T03:14:16.626996+0000","last_clean":"2024-03-12T03:14:16.626996+0000","last_became_active":"2024-03-12T03:13:44.946095+0000","last_became_peered":"2024-03-12T03:13:44.946095+0000","last_unstale":"2024-03-12T03:14:16.626996+0000","last_undegraded":"2024-03-12T03:14:16.626996+0000","last_fullsized":"2024-03-12T03:14:16.626996+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:43:08.958607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.998743+0000","last_change":"2024-03-12T03:13:42.942423+0000","last_active":"2024-03-12T03:14:16.998743+0000","last_peered":"2024-03-12T03:14:16.998743+0000","last_clean":"2024-03-12T03:14:16.998743+0000","last_became_active":"2024-03-12T03:13:42.942038+0000","last_became_peered":"2024-03-12T03:13:42.942038+0000","last_unstale":"2024-03-12T03:14:16.998743+0000","last_undegraded":"2024-03-12T03:14:16.998743+0000","last_fullsized":"2024-03-12T03:14:16.998743+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:41:46.231848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999873+0000","last_change":"2024-03-12T03:13:38.923824+0000","last_active":"2024-03-12T03:14:16.999873+0000","last_peered":"2024-03-12T03:14:16.999873+0000","last_clean":"2024-03-12T03:14:16.999873+0000","last_became_active":"2024-03-12T03:13:38.923440+0000","last_became_peered":"2024-03-12T03:13:38.923440+0000","last_unstale":"2024-03-12T03:14:16.999873+0000","last_undegraded":"2024-03-12T03:14:16.999873+0000","last_fullsized":"2024-03-12T03:14:16.999873+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:22:06.378441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":35,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509630+0000","last_change":"2024-03-12T03:13:40.923766+0000","last_active":"2024-03-12T03:14:16.509630+0000","last_peered":"2024-03-12T03:14:16.509630+0000","last_clean":"2024-03-12T03:14:16.509630+0000","last_became_active":"2024-03-12T03:13:40.923460+0000","last_became_peered":"2024-03-12T03:13:40.923460+0000","last_unstale":"2024-03-12T03:14:16.509630+0000","last_undegraded":"2024-03-12T03:14:16.509630+0000","last_fullsized":"2024-03-12T03:14:16.509630+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:37:06.539360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510145+0000","last_change":"2024-03-12T03:13:44.998732+0000","last_active":"2024-03-12T03:14:16.510145+0000","last_peered":"2024-03-12T03:14:16.510145+0000","last_clean":"2024-03-12T03:14:16.510145+0000","last_became_active":"2024-03-12T03:13:44.997545+0000","last_became_peered":"2024-03-12T03:13:44.997545+0000","last_unstale":"2024-03-12T03:14:16.510145+0000","last_undegraded":"2024-03-12T03:14:16.510145+0000","last_fullsized":"2024-03-12T03:14:16.510145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:27:31.542973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627165+0000","last_change":"2024-03-12T03:13:42.944935+0000","last_active":"2024-03-12T03:14:16.627165+0000","last_peered":"2024-03-12T03:14:16.627165+0000","last_clean":"2024-03-12T03:14:16.627165+0000","last_became_active":"2024-03-12T03:13:42.944626+0000","last_became_peered":"2024-03-12T03:13:42.944626+0000","last_unstale":"2024-03-12T03:14:16.627165+0000","last_undegraded":"2024-03-12T03:14:16.627165+0000","last_fullsized":"2024-03-12T03:14:16.627165+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:31:46.968518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615884+0000","last_change":"2024-03-12T03:13:38.920375+0000","last_active":"2024-03-12T03:14:16.615884+0000","last_peered":"2024-03-12T03:14:16.615884+0000","last_clean":"2024-03-12T03:14:16.615884+0000","last_became_active":"2024-03-12T03:13:38.920074+0000","last_became_peered":"2024-03-12T03:13:38.920074+0000","last_unstale":"2024-03-12T03:14:16.615884+0000","last_undegraded":"2024-03-12T03:14:16.615884+0000","last_fullsized":"2024-03-12T03:14:16.615884+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:23:34.980491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'17","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.513207+0000","last_change":"2024-03-12T03:13:41.097565+0000","last_active":"2024-03-12T03:14:16.513207+0000","last_peered":"2024-03-12T03:14:16.513207+0000","last_clean":"2024-03-12T03:14:16.513207+0000","last_became_active":"2024-03-12T03:13:41.097163+0000","last_became_peered":"2024-03-12T03:13:41.097163+0000","last_unstale":"2024-03-12T03:14:16.513207+0000","last_undegraded":"2024-03-12T03:14:16.513207+0000","last_fullsized":"2024-03-12T03:14:16.513207+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:36.405442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999867+0000","last_change":"2024-03-12T03:13:44.947369+0000","last_active":"2024-03-12T03:14:16.999867+0000","last_peered":"2024-03-12T03:14:16.999867+0000","last_clean":"2024-03-12T03:14:16.999867+0000","last_became_active":"2024-03-12T03:13:44.947062+0000","last_became_peered":"2024-03-12T03:13:44.947062+0000","last_unstale":"2024-03-12T03:14:16.999867+0000","last_undegraded":"2024-03-12T03:14:16.999867+0000","last_fullsized":"2024-03-12T03:14:16.999867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:15:30.842885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510399+0000","last_change":"2024-03-12T03:13:42.939145+0000","last_active":"2024-03-12T03:14:16.510399+0000","last_peered":"2024-03-12T03:14:16.510399+0000","last_clean":"2024-03-12T03:14:16.510399+0000","last_became_active":"2024-03-12T03:13:42.938795+0000","last_became_peered":"2024-03-12T03:13:42.938795+0000","last_unstale":"2024-03-12T03:14:16.510399+0000","last_undegraded":"2024-03-12T03:14:16.510399+0000","last_fullsized":"2024-03-12T03:14:16.510399+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:09:01.041567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627374+0000","last_change":"2024-03-12T03:13:38.924544+0000","last_active":"2024-03-12T03:14:16.627374+0000","last_peered":"2024-03-12T03:14:16.627374+0000","last_clean":"2024-03-12T03:14:16.627374+0000","last_became_active":"2024-03-12T03:13:38.924274+0000","last_became_peered":"2024-03-12T03:13:38.924274+0000","last_unstale":"2024-03-12T03:14:16.627374+0000","last_undegraded":"2024-03-12T03:14:16.627374+0000","last_fullsized":"2024-03-12T03:14:16.627374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:43:37.403267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509577+0000","last_change":"2024-03-12T03:13:40.926321+0000","last_active":"2024-03-12T03:14:16.509577+0000","last_peered":"2024-03-12T03:14:16.509577+0000","last_clean":"2024-03-12T03:14:16.509577+0000","last_became_active":"2024-03-12T03:13:40.924605+0000","last_became_peered":"2024-03-12T03:13:40.924605+0000","last_unstale":"2024-03-12T03:14:16.509577+0000","last_undegraded":"2024-03-12T03:14:16.509577+0000","last_fullsized":"2024-03-12T03:14:16.509577+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:55:43.295784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.512939+0000","last_change":"2024-03-12T03:13:44.946755+0000","last_active":"2024-03-12T03:14:16.512939+0000","last_peered":"2024-03-12T03:14:16.512939+0000","last_clean":"2024-03-12T03:14:16.512939+0000","last_became_active":"2024-03-12T03:13:44.946404+0000","last_became_peered":"2024-03-12T03:13:44.946404+0000","last_unstale":"2024-03-12T03:14:16.512939+0000","last_undegraded":"2024-03-12T03:14:16.512939+0000","last_fullsized":"2024-03-12T03:14:16.512939+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:47:34.387479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509390+0000","last_change":"2024-03-12T03:13:42.935816+0000","last_active":"2024-03-12T03:14:16.509390+0000","last_peered":"2024-03-12T03:14:16.509390+0000","last_clean":"2024-03-12T03:14:16.509390+0000","last_became_active":"2024-03-12T03:13:42.935482+0000","last_became_peered":"2024-03-12T03:13:42.935482+0000","last_unstale":"2024-03-12T03:14:16.509390+0000","last_undegraded":"2024-03-12T03:14:16.509390+0000","last_fullsized":"2024-03-12T03:14:16.509390+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:43:08.609559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510989+0000","last_change":"2024-03-12T03:13:38.926229+0000","last_active":"2024-03-12T03:14:16.510989+0000","last_peered":"2024-03-12T03:14:16.510989+0000","last_clean":"2024-03-12T03:14:16.510989+0000","last_became_active":"2024-03-12T03:13:38.925917+0000","last_became_peered":"2024-03-12T03:13:38.925917+0000","last_unstale":"2024-03-12T03:14:16.510989+0000","last_undegraded":"2024-03-12T03:14:16.510989+0000","last_fullsized":"2024-03-12T03:14:16.510989+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:07:13.015269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510798+0000","last_change":"2024-03-12T03:13:40.924138+0000","last_active":"2024-03-12T03:14:16.510798+0000","last_peered":"2024-03-12T03:14:16.510798+0000","last_clean":"2024-03-12T03:14:16.510798+0000","last_became_active":"2024-03-12T03:13:40.923266+0000","last_became_peered":"2024-03-12T03:13:40.923266+0000","last_unstale":"2024-03-12T03:14:16.510798+0000","last_undegraded":"2024-03-12T03:14:16.510798+0000","last_fullsized":"2024-03-12T03:14:16.510798+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:08:29.879033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999428+0000","last_change":"2024-03-12T03:13:44.950883+0000","last_active":"2024-03-12T03:14:16.999428+0000","last_peered":"2024-03-12T03:14:16.999428+0000","last_clean":"2024-03-12T03:14:16.999428+0000","last_became_active":"2024-03-12T03:13:44.950518+0000","last_became_peered":"2024-03-12T03:13:44.950518+0000","last_unstale":"2024-03-12T03:14:16.999428+0000","last_undegraded":"2024-03-12T03:14:16.999428+0000","last_fullsized":"2024-03-12T03:14:16.999428+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:03:18.832128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.982035+0000","last_change":"2024-03-12T03:13:42.937176+0000","last_active":"2024-03-12T03:14:53.982035+0000","last_peered":"2024-03-12T03:14:53.982035+0000","last_clean":"2024-03-12T03:14:53.982035+0000","last_became_active":"2024-03-12T03:13:42.936829+0000","last_became_peered":"2024-03-12T03:13:42.936829+0000","last_unstale":"2024-03-12T03:14:53.982035+0000","last_undegraded":"2024-03-12T03:14:53.982035+0000","last_fullsized":"2024-03-12T03:14:53.982035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:00:14.343096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627210+0000","last_change":"2024-03-12T03:13:38.929956+0000","last_active":"2024-03-12T03:14:16.627210+0000","last_peered":"2024-03-12T03:14:16.627210+0000","last_clean":"2024-03-12T03:14:16.627210+0000","last_became_active":"2024-03-12T03:13:38.929652+0000","last_became_peered":"2024-03-12T03:13:38.929652+0000","last_unstale":"2024-03-12T03:14:16.627210+0000","last_undegraded":"2024-03-12T03:14:16.627210+0000","last_fullsized":"2024-03-12T03:14:16.627210+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:55:18.867286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'22","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510770+0000","last_change":"2024-03-12T03:13:41.097345+0000","last_active":"2024-03-12T03:14:16.510770+0000","last_peered":"2024-03-12T03:14:16.510770+0000","last_clean":"2024-03-12T03:14:16.510770+0000","last_became_active":"2024-03-12T03:13:41.097015+0000","last_became_peered":"2024-03-12T03:13:41.097015+0000","last_unstale":"2024-03-12T03:14:16.510770+0000","last_undegraded":"2024-03-12T03:14:16.510770+0000","last_fullsized":"2024-03-12T03:14:16.510770+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:15:35.016446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.511292+0000","last_change":"2024-03-12T03:13:44.947715+0000","last_active":"2024-03-12T03:14:16.511292+0000","last_peered":"2024-03-12T03:14:16.511292+0000","last_clean":"2024-03-12T03:14:16.511292+0000","last_became_active":"2024-03-12T03:13:44.947374+0000","last_became_peered":"2024-03-12T03:13:44.947374+0000","last_unstale":"2024-03-12T03:14:16.511292+0000","last_undegraded":"2024-03-12T03:14:16.511292+0000","last_fullsized":"2024-03-12T03:14:16.511292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:57:22.608130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.982013+0000","last_change":"2024-03-12T03:13:42.937374+0000","last_active":"2024-03-12T03:14:53.982013+0000","last_peered":"2024-03-12T03:14:53.982013+0000","last_clean":"2024-03-12T03:14:53.982013+0000","last_became_active":"2024-03-12T03:13:42.936444+0000","last_became_peered":"2024-03-12T03:13:42.936444+0000","last_unstale":"2024-03-12T03:14:53.982013+0000","last_undegraded":"2024-03-12T03:14:53.982013+0000","last_fullsized":"2024-03-12T03:14:53.982013+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:47:01.032256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999091+0000","last_change":"2024-03-12T03:13:38.924248+0000","last_active":"2024-03-12T03:14:16.999091+0000","last_peered":"2024-03-12T03:14:16.999091+0000","last_clean":"2024-03-12T03:14:16.999091+0000","last_became_active":"2024-03-12T03:13:38.923957+0000","last_became_peered":"2024-03-12T03:13:38.923957+0000","last_unstale":"2024-03-12T03:14:16.999091+0000","last_undegraded":"2024-03-12T03:14:16.999091+0000","last_fullsized":"2024-03-12T03:14:16.999091+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:26:00.742528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'15","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615965+0000","last_change":"2024-03-12T03:13:41.097730+0000","last_active":"2024-03-12T03:14:16.615965+0000","last_peered":"2024-03-12T03:14:16.615965+0000","last_clean":"2024-03-12T03:14:16.615965+0000","last_became_active":"2024-03-12T03:13:41.097368+0000","last_became_peered":"2024-03-12T03:13:41.097368+0000","last_unstale":"2024-03-12T03:14:16.615965+0000","last_undegraded":"2024-03-12T03:14:16.615965+0000","last_fullsized":"2024-03-12T03:14:16.615965+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:29:19.105913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.625206+0000","last_change":"2024-03-12T03:13:44.946631+0000","last_active":"2024-03-12T03:14:16.625206+0000","last_peered":"2024-03-12T03:14:16.625206+0000","last_clean":"2024-03-12T03:14:16.625206+0000","last_became_active":"2024-03-12T03:13:44.946314+0000","last_became_peered":"2024-03-12T03:13:44.946314+0000","last_unstale":"2024-03-12T03:14:16.625206+0000","last_undegraded":"2024-03-12T03:14:16.625206+0000","last_fullsized":"2024-03-12T03:14:16.625206+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:17:12.123391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.512342+0000","last_change":"2024-03-12T03:13:42.938275+0000","last_active":"2024-03-12T03:14:16.512342+0000","last_peered":"2024-03-12T03:14:16.512342+0000","last_clean":"2024-03-12T03:14:16.512342+0000","last_became_active":"2024-03-12T03:13:42.937817+0000","last_became_peered":"2024-03-12T03:13:42.937817+0000","last_unstale":"2024-03-12T03:14:16.512342+0000","last_undegraded":"2024-03-12T03:14:16.512342+0000","last_fullsized":"2024-03-12T03:14:16.512342+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:43:41.883229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627057+0000","last_change":"2024-03-12T03:13:38.921312+0000","last_active":"2024-03-12T03:14:16.627057+0000","last_peered":"2024-03-12T03:14:16.627057+0000","last_clean":"2024-03-12T03:14:16.627057+0000","last_became_active":"2024-03-12T03:13:38.921028+0000","last_became_peered":"2024-03-12T03:13:38.921028+0000","last_unstale":"2024-03-12T03:14:16.627057+0000","last_undegraded":"2024-03-12T03:14:16.627057+0000","last_fullsized":"2024-03-12T03:14:16.627057+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:47:47.532087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'23","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000230+0000","last_change":"2024-03-12T03:13:41.098535+0000","last_active":"2024-03-12T03:14:17.000230+0000","last_peered":"2024-03-12T03:14:17.000230+0000","last_clean":"2024-03-12T03:14:17.000230+0000","last_became_active":"2024-03-12T03:13:41.098250+0000","last_became_peered":"2024-03-12T03:13:41.098250+0000","last_unstale":"2024-03-12T03:14:17.000230+0000","last_undegraded":"2024-03-12T03:14:17.000230+0000","last_fullsized":"2024-03-12T03:14:17.000230+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:10:03.036836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627273+0000","last_change":"2024-03-12T03:13:44.997523+0000","last_active":"2024-03-12T03:14:16.627273+0000","last_peered":"2024-03-12T03:14:16.627273+0000","last_clean":"2024-03-12T03:14:16.627273+0000","last_became_active":"2024-03-12T03:13:44.997227+0000","last_became_peered":"2024-03-12T03:13:44.997227+0000","last_unstale":"2024-03-12T03:14:16.627273+0000","last_undegraded":"2024-03-12T03:14:16.627273+0000","last_fullsized":"2024-03-12T03:14:16.627273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:05:08.063765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615919+0000","last_change":"2024-03-12T03:13:42.932987+0000","last_active":"2024-03-12T03:14:16.615919+0000","last_peered":"2024-03-12T03:14:16.615919+0000","last_clean":"2024-03-12T03:14:16.615919+0000","last_became_active":"2024-03-12T03:13:42.932660+0000","last_became_peered":"2024-03-12T03:13:42.932660+0000","last_unstale":"2024-03-12T03:14:16.615919+0000","last_undegraded":"2024-03-12T03:14:16.615919+0000","last_fullsized":"2024-03-12T03:14:16.615919+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:20.436208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.998805+0000","last_change":"2024-03-12T03:13:38.918748+0000","last_active":"2024-03-12T03:14:16.998805+0000","last_peered":"2024-03-12T03:14:16.998805+0000","last_clean":"2024-03-12T03:14:16.998805+0000","last_became_active":"2024-03-12T03:13:38.918453+0000","last_became_peered":"2024-03-12T03:13:38.918453+0000","last_unstale":"2024-03-12T03:14:16.998805+0000","last_undegraded":"2024-03-12T03:14:16.998805+0000","last_fullsized":"2024-03-12T03:14:16.998805+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:36:56.790011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'12","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.512854+0000","last_change":"2024-03-12T03:13:41.097854+0000","last_active":"2024-03-12T03:14:16.512854+0000","last_peered":"2024-03-12T03:14:16.512854+0000","last_clean":"2024-03-12T03:14:16.512854+0000","last_became_active":"2024-03-12T03:13:41.097410+0000","last_became_peered":"2024-03-12T03:13:41.097410+0000","last_unstale":"2024-03-12T03:14:16.512854+0000","last_undegraded":"2024-03-12T03:14:16.512854+0000","last_fullsized":"2024-03-12T03:14:16.512854+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:13.635492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615622+0000","last_change":"2024-03-12T03:13:44.997634+0000","last_active":"2024-03-12T03:14:16.615622+0000","last_peered":"2024-03-12T03:14:16.615622+0000","last_clean":"2024-03-12T03:14:16.615622+0000","last_became_active":"2024-03-12T03:13:44.997278+0000","last_became_peered":"2024-03-12T03:13:44.997278+0000","last_unstale":"2024-03-12T03:14:16.615622+0000","last_undegraded":"2024-03-12T03:14:16.615622+0000","last_fullsized":"2024-03-12T03:14:16.615622+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:12:19.740465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627596+0000","last_change":"2024-03-12T03:13:38.929873+0000","last_active":"2024-03-12T03:14:16.627596+0000","last_peered":"2024-03-12T03:14:16.627596+0000","last_clean":"2024-03-12T03:14:16.627596+0000","last_became_active":"2024-03-12T03:13:38.929583+0000","last_became_peered":"2024-03-12T03:13:38.929583+0000","last_unstale":"2024-03-12T03:14:16.627596+0000","last_undegraded":"2024-03-12T03:14:16.627596+0000","last_fullsized":"2024-03-12T03:14:16.627596+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:49:15.740169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'27","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615373+0000","last_change":"2024-03-12T03:13:41.097793+0000","last_active":"2024-03-12T03:14:16.615373+0000","last_peered":"2024-03-12T03:14:16.615373+0000","last_clean":"2024-03-12T03:14:16.615373+0000","last_became_active":"2024-03-12T03:13:41.097430+0000","last_became_peered":"2024-03-12T03:13:41.097430+0000","last_unstale":"2024-03-12T03:14:16.615373+0000","last_undegraded":"2024-03-12T03:14:16.615373+0000","last_fullsized":"2024-03-12T03:14:16.615373+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:36.711439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.982110+0000","last_change":"2024-03-12T03:13:42.943578+0000","last_active":"2024-03-12T03:14:53.982110+0000","last_peered":"2024-03-12T03:14:53.982110+0000","last_clean":"2024-03-12T03:14:53.982110+0000","last_became_active":"2024-03-12T03:13:42.943319+0000","last_became_peered":"2024-03-12T03:13:42.943319+0000","last_unstale":"2024-03-12T03:14:53.982110+0000","last_undegraded":"2024-03-12T03:14:53.982110+0000","last_fullsized":"2024-03-12T03:14:53.982110+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:24:24.897970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510234+0000","last_change":"2024-03-12T03:13:44.943749+0000","last_active":"2024-03-12T03:14:16.510234+0000","last_peered":"2024-03-12T03:14:16.510234+0000","last_clean":"2024-03-12T03:14:16.510234+0000","last_became_active":"2024-03-12T03:13:44.943350+0000","last_became_peered":"2024-03-12T03:13:44.943350+0000","last_unstale":"2024-03-12T03:14:16.510234+0000","last_undegraded":"2024-03-12T03:14:16.510234+0000","last_fullsized":"2024-03-12T03:14:16.510234+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:04:52.403194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627494+0000","last_change":"2024-03-12T03:13:38.930076+0000","last_active":"2024-03-12T03:14:16.627494+0000","last_peered":"2024-03-12T03:14:16.627494+0000","last_clean":"2024-03-12T03:14:16.627494+0000","last_became_active":"2024-03-12T03:13:38.929785+0000","last_became_peered":"2024-03-12T03:13:38.929785+0000","last_unstale":"2024-03-12T03:14:16.627494+0000","last_undegraded":"2024-03-12T03:14:16.627494+0000","last_fullsized":"2024-03-12T03:14:16.627494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:31:45.008883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'24","reported_seq":98,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510370+0000","last_change":"2024-03-12T03:14:04.018882+0000","last_active":"2024-03-12T03:14:16.510370+0000","last_peered":"2024-03-12T03:14:16.510370+0000","last_clean":"2024-03-12T03:14:16.510370+0000","last_became_active":"2024-03-12T03:14:04.005976+0000","last_became_peered":"2024-03-12T03:14:04.005976+0000","last_unstale":"2024-03-12T03:14:16.510370+0000","last_undegraded":"2024-03-12T03:14:16.510370+0000","last_fullsized":"2024-03-12T03:14:16.510370+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:39:52.513477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.623642+0000","last_change":"2024-03-12T03:13:42.930881+0000","last_active":"2024-03-12T03:14:16.623642+0000","last_peered":"2024-03-12T03:14:16.623642+0000","last_clean":"2024-03-12T03:14:16.623642+0000","last_became_active":"2024-03-12T03:13:42.930558+0000","last_became_peered":"2024-03-12T03:13:42.930558+0000","last_unstale":"2024-03-12T03:14:16.623642+0000","last_undegraded":"2024-03-12T03:14:16.623642+0000","last_fullsized":"2024-03-12T03:14:16.623642+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:01.690017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.626850+0000","last_change":"2024-03-12T03:13:44.946239+0000","last_active":"2024-03-12T03:14:16.626850+0000","last_peered":"2024-03-12T03:14:16.626850+0000","last_clean":"2024-03-12T03:14:16.626850+0000","last_became_active":"2024-03-12T03:13:44.945891+0000","last_became_peered":"2024-03-12T03:13:44.945891+0000","last_unstale":"2024-03-12T03:14:16.626850+0000","last_undegraded":"2024-03-12T03:14:16.626850+0000","last_fullsized":"2024-03-12T03:14:16.626850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:06:53.343828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.624145+0000","last_change":"2024-03-12T03:13:42.933257+0000","last_active":"2024-03-12T03:14:16.624145+0000","last_peered":"2024-03-12T03:14:16.624145+0000","last_clean":"2024-03-12T03:14:16.624145+0000","last_became_active":"2024-03-12T03:13:42.932919+0000","last_became_peered":"2024-03-12T03:13:42.932919+0000","last_unstale":"2024-03-12T03:14:16.624145+0000","last_undegraded":"2024-03-12T03:14:16.624145+0000","last_fullsized":"2024-03-12T03:14:16.624145+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:22.916168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627680+0000","last_change":"2024-03-12T03:13:38.927221+0000","last_active":"2024-03-12T03:14:16.627680+0000","last_peered":"2024-03-12T03:14:16.627680+0000","last_clean":"2024-03-12T03:14:16.627680+0000","last_became_active":"2024-03-12T03:13:38.926918+0000","last_became_peered":"2024-03-12T03:13:38.926918+0000","last_unstale":"2024-03-12T03:14:16.627680+0000","last_undegraded":"2024-03-12T03:14:16.627680+0000","last_fullsized":"2024-03-12T03:14:16.627680+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:44:56.107368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'25","reported_seq":100,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999413+0000","last_change":"2024-03-12T03:14:04.191395+0000","last_active":"2024-03-12T03:14:16.999413+0000","last_peered":"2024-03-12T03:14:16.999413+0000","last_clean":"2024-03-12T03:14:16.999413+0000","last_became_active":"2024-03-12T03:14:04.180406+0000","last_became_peered":"2024-03-12T03:14:04.180406+0000","last_unstale":"2024-03-12T03:14:16.999413+0000","last_undegraded":"2024-03-12T03:14:16.999413+0000","last_fullsized":"2024-03-12T03:14:16.999413+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:01:27.932066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999018+0000","last_change":"2024-03-12T03:13:44.948367+0000","last_active":"2024-03-12T03:14:16.999018+0000","last_peered":"2024-03-12T03:14:16.999018+0000","last_clean":"2024-03-12T03:14:16.999018+0000","last_became_active":"2024-03-12T03:13:44.948026+0000","last_became_peered":"2024-03-12T03:13:44.948026+0000","last_unstale":"2024-03-12T03:14:16.999018+0000","last_undegraded":"2024-03-12T03:14:16.999018+0000","last_fullsized":"2024-03-12T03:14:16.999018+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:50:55.995044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.623809+0000","last_change":"2024-03-12T03:13:42.930758+0000","last_active":"2024-03-12T03:14:16.623809+0000","last_peered":"2024-03-12T03:14:16.623809+0000","last_clean":"2024-03-12T03:14:16.623809+0000","last_became_active":"2024-03-12T03:13:42.930418+0000","last_became_peered":"2024-03-12T03:13:42.930418+0000","last_unstale":"2024-03-12T03:14:16.623809+0000","last_undegraded":"2024-03-12T03:14:16.623809+0000","last_fullsized":"2024-03-12T03:14:16.623809+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:54:48.011915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615673+0000","last_change":"2024-03-12T03:13:38.923640+0000","last_active":"2024-03-12T03:14:16.615673+0000","last_peered":"2024-03-12T03:14:16.615673+0000","last_clean":"2024-03-12T03:14:16.615673+0000","last_became_active":"2024-03-12T03:13:38.923299+0000","last_became_peered":"2024-03-12T03:13:38.923299+0000","last_unstale":"2024-03-12T03:14:16.615673+0000","last_undegraded":"2024-03-12T03:14:16.615673+0000","last_fullsized":"2024-03-12T03:14:16.615673+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:33:53.613151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'15","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.001049+0000","last_change":"2024-03-12T03:13:40.933773+0000","last_active":"2024-03-12T03:14:17.001049+0000","last_peered":"2024-03-12T03:14:17.001049+0000","last_clean":"2024-03-12T03:14:17.001049+0000","last_became_active":"2024-03-12T03:13:40.933262+0000","last_became_peered":"2024-03-12T03:13:40.933262+0000","last_unstale":"2024-03-12T03:14:17.001049+0000","last_undegraded":"2024-03-12T03:14:17.001049+0000","last_fullsized":"2024-03-12T03:14:17.001049+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:28:49.669840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.511458+0000","last_change":"2024-03-12T03:13:44.944810+0000","last_active":"2024-03-12T03:14:16.511458+0000","last_peered":"2024-03-12T03:14:16.511458+0000","last_clean":"2024-03-12T03:14:16.511458+0000","last_became_active":"2024-03-12T03:13:44.943384+0000","last_became_peered":"2024-03-12T03:13:44.943384+0000","last_unstale":"2024-03-12T03:14:16.511458+0000","last_undegraded":"2024-03-12T03:14:16.511458+0000","last_fullsized":"2024-03-12T03:14:16.511458+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:17.957042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615461+0000","last_change":"2024-03-12T03:13:41.921832+0000","last_active":"2024-03-12T03:14:16.615461+0000","last_peered":"2024-03-12T03:14:16.615461+0000","last_clean":"2024-03-12T03:14:16.615461+0000","last_became_active":"2024-03-12T03:13:37.916764+0000","last_became_peered":"2024-03-12T03:13:37.916764+0000","last_unstale":"2024-03-12T03:14:16.615461+0000","last_undegraded":"2024-03-12T03:14:16.615461+0000","last_fullsized":"2024-03-12T03:14:16.615461+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:36.900834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:36.900834+0000","last_clean_scrub_stamp":"2024-03-12T03:13:36.900834+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:01.955380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035682550000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510416+0000","last_change":"2024-03-12T03:13:38.921875+0000","last_active":"2024-03-12T03:14:16.510416+0000","last_peered":"2024-03-12T03:14:16.510416+0000","last_clean":"2024-03-12T03:14:16.510416+0000","last_became_active":"2024-03-12T03:13:38.921588+0000","last_became_peered":"2024-03-12T03:13:38.921588+0000","last_unstale":"2024-03-12T03:14:16.510416+0000","last_undegraded":"2024-03-12T03:14:16.510416+0000","last_fullsized":"2024-03-12T03:14:16.510416+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:07:57.296831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'34","reported_seq":106,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.981737+0000","last_change":"2024-03-12T03:13:40.929007+0000","last_active":"2024-03-12T03:14:53.981737+0000","last_peered":"2024-03-12T03:14:53.981737+0000","last_clean":"2024-03-12T03:14:53.981737+0000","last_became_active":"2024-03-12T03:13:40.928584+0000","last_became_peered":"2024-03-12T03:13:40.928584+0000","last_unstale":"2024-03-12T03:14:53.981737+0000","last_undegraded":"2024-03-12T03:14:53.981737+0000","last_fullsized":"2024-03-12T03:14:53.981737+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:46:43.916913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999906+0000","last_change":"2024-03-12T03:13:42.930916+0000","last_active":"2024-03-12T03:14:16.999906+0000","last_peered":"2024-03-12T03:14:16.999906+0000","last_clean":"2024-03-12T03:14:16.999906+0000","last_became_active":"2024-03-12T03:13:42.930028+0000","last_became_peered":"2024-03-12T03:13:42.930028+0000","last_unstale":"2024-03-12T03:14:16.999906+0000","last_undegraded":"2024-03-12T03:14:16.999906+0000","last_fullsized":"2024-03-12T03:14:16.999906+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:59:16.700540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627860+0000","last_change":"2024-03-12T03:13:44.998674+0000","last_active":"2024-03-12T03:14:16.627860+0000","last_peered":"2024-03-12T03:14:16.627860+0000","last_clean":"2024-03-12T03:14:16.627860+0000","last_became_active":"2024-03-12T03:13:44.998360+0000","last_became_peered":"2024-03-12T03:13:44.998360+0000","last_unstale":"2024-03-12T03:14:16.627860+0000","last_undegraded":"2024-03-12T03:14:16.627860+0000","last_fullsized":"2024-03-12T03:14:16.627860+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:55:34.393299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.508782+0000","last_change":"2024-03-12T03:13:38.919604+0000","last_active":"2024-03-12T03:14:16.508782+0000","last_peered":"2024-03-12T03:14:16.508782+0000","last_clean":"2024-03-12T03:14:16.508782+0000","last_became_active":"2024-03-12T03:13:38.918065+0000","last_became_peered":"2024-03-12T03:13:38.918065+0000","last_unstale":"2024-03-12T03:14:16.508782+0000","last_undegraded":"2024-03-12T03:14:16.508782+0000","last_fullsized":"2024-03-12T03:14:16.508782+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:33:12.232739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.624200+0000","last_change":"2024-03-12T03:13:41.916325+0000","last_active":"2024-03-12T03:14:16.624200+0000","last_peered":"2024-03-12T03:14:16.624200+0000","last_clean":"2024-03-12T03:14:16.624200+0000","last_became_active":"2024-03-12T03:13:37.917610+0000","last_became_peered":"2024-03-12T03:13:37.917610+0000","last_unstale":"2024-03-12T03:14:16.624200+0000","last_undegraded":"2024-03-12T03:14:16.624200+0000","last_fullsized":"2024-03-12T03:14:16.624200+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:36.900834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:36.900834+0000","last_clean_scrub_stamp":"2024-03-12T03:13:36.900834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:53:54.666848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055126199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'19","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.511530+0000","last_change":"2024-03-12T03:13:40.927880+0000","last_active":"2024-03-12T03:14:16.511530+0000","last_peered":"2024-03-12T03:14:16.511530+0000","last_clean":"2024-03-12T03:14:16.511530+0000","last_became_active":"2024-03-12T03:13:40.924923+0000","last_became_peered":"2024-03-12T03:13:40.924923+0000","last_unstale":"2024-03-12T03:14:16.511530+0000","last_undegraded":"2024-03-12T03:14:16.511530+0000","last_fullsized":"2024-03-12T03:14:16.511530+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:51:51.830745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.624338+0000","last_change":"2024-03-12T03:13:42.935113+0000","last_active":"2024-03-12T03:14:16.624338+0000","last_peered":"2024-03-12T03:14:16.624338+0000","last_clean":"2024-03-12T03:14:16.624338+0000","last_became_active":"2024-03-12T03:13:42.934797+0000","last_became_peered":"2024-03-12T03:13:42.934797+0000","last_unstale":"2024-03-12T03:14:16.624338+0000","last_undegraded":"2024-03-12T03:14:16.624338+0000","last_fullsized":"2024-03-12T03:14:16.624338+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:49:17.115408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615886+0000","last_change":"2024-03-12T03:13:44.940697+0000","last_active":"2024-03-12T03:14:16.615886+0000","last_peered":"2024-03-12T03:14:16.615886+0000","last_clean":"2024-03-12T03:14:16.615886+0000","last_became_active":"2024-03-12T03:13:44.940391+0000","last_became_peered":"2024-03-12T03:13:44.940391+0000","last_unstale":"2024-03-12T03:14:16.615886+0000","last_undegraded":"2024-03-12T03:14:16.615886+0000","last_fullsized":"2024-03-12T03:14:16.615886+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:48:46.696099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"72'128","reported_seq":278,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:18.628450+0000","last_change":"2024-03-12T03:13:28.492572+0000","last_active":"2024-03-12T03:14:18.628450+0000","last_peered":"2024-03-12T03:14:18.628450+0000","last_clean":"2024-03-12T03:14:18.628450+0000","last_became_active":"2024-03-12T03:13:28.492056+0000","last_became_peered":"2024-03-12T03:13:28.492056+0000","last_unstale":"2024-03-12T03:14:18.628450+0000","last_undegraded":"2024-03-12T03:14:18.628450+0000","last_fullsized":"2024-03-12T03:14:18.628450+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:11:21.093480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:11:21.093480+0000","last_clean_scrub_stamp":"2024-03-12T03:11:21.093480+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:59:57.354004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509972+0000","last_change":"2024-03-12T03:13:38.927650+0000","last_active":"2024-03-12T03:14:16.509972+0000","last_peered":"2024-03-12T03:14:16.509972+0000","last_clean":"2024-03-12T03:14:16.509972+0000","last_became_active":"2024-03-12T03:13:38.927310+0000","last_became_peered":"2024-03-12T03:13:38.927310+0000","last_unstale":"2024-03-12T03:14:16.509972+0000","last_undegraded":"2024-03-12T03:14:16.509972+0000","last_fullsized":"2024-03-12T03:14:16.509972+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:38:20.477500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.982205+0000","last_change":"2024-03-12T03:13:41.098065+0000","last_active":"2024-03-12T03:14:53.982205+0000","last_peered":"2024-03-12T03:14:53.982205+0000","last_clean":"2024-03-12T03:14:53.982205+0000","last_became_active":"2024-03-12T03:13:41.097186+0000","last_became_peered":"2024-03-12T03:13:41.097186+0000","last_unstale":"2024-03-12T03:14:53.982205+0000","last_undegraded":"2024-03-12T03:14:53.982205+0000","last_fullsized":"2024-03-12T03:14:53.982205+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:42:22.670132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627755+0000","last_change":"2024-03-12T03:13:42.940542+0000","last_active":"2024-03-12T03:14:16.627755+0000","last_peered":"2024-03-12T03:14:16.627755+0000","last_clean":"2024-03-12T03:14:16.627755+0000","last_became_active":"2024-03-12T03:13:42.940245+0000","last_became_peered":"2024-03-12T03:13:42.940245+0000","last_unstale":"2024-03-12T03:14:16.627755+0000","last_undegraded":"2024-03-12T03:14:16.627755+0000","last_fullsized":"2024-03-12T03:14:16.627755+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:01.095734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510001+0000","last_change":"2024-03-12T03:13:44.949096+0000","last_active":"2024-03-12T03:14:16.510001+0000","last_peered":"2024-03-12T03:14:16.510001+0000","last_clean":"2024-03-12T03:14:16.510001+0000","last_became_active":"2024-03-12T03:13:44.948108+0000","last_became_peered":"2024-03-12T03:13:44.948108+0000","last_unstale":"2024-03-12T03:14:16.510001+0000","last_undegraded":"2024-03-12T03:14:16.510001+0000","last_fullsized":"2024-03-12T03:14:16.510001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:04:58.946629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999310+0000","last_change":"2024-03-12T03:13:38.924200+0000","last_active":"2024-03-12T03:14:16.999310+0000","last_peered":"2024-03-12T03:14:16.999310+0000","last_clean":"2024-03-12T03:14:16.999310+0000","last_became_active":"2024-03-12T03:13:38.923882+0000","last_became_peered":"2024-03-12T03:13:38.923882+0000","last_unstale":"2024-03-12T03:14:16.999310+0000","last_undegraded":"2024-03-12T03:14:16.999310+0000","last_fullsized":"2024-03-12T03:14:16.999310+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:29:21.228313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":122,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:15:00.812449+0000","last_change":"2024-03-12T03:13:41.922084+0000","last_active":"2024-03-12T03:15:00.812449+0000","last_peered":"2024-03-12T03:15:00.812449+0000","last_clean":"2024-03-12T03:15:00.812449+0000","last_became_active":"2024-03-12T03:13:37.917516+0000","last_became_peered":"2024-03-12T03:13:37.917516+0000","last_unstale":"2024-03-12T03:15:00.812449+0000","last_undegraded":"2024-03-12T03:15:00.812449+0000","last_fullsized":"2024-03-12T03:15:00.812449+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:36.900834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:36.900834+0000","last_clean_scrub_stamp":"2024-03-12T03:13:36.900834+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:30:40.218046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000451154,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":77,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":47,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000071+0000","last_change":"2024-03-12T03:13:40.932547+0000","last_active":"2024-03-12T03:14:17.000071+0000","last_peered":"2024-03-12T03:14:17.000071+0000","last_clean":"2024-03-12T03:14:17.000071+0000","last_became_active":"2024-03-12T03:13:40.932123+0000","last_became_peered":"2024-03-12T03:13:40.932123+0000","last_unstale":"2024-03-12T03:14:17.000071+0000","last_undegraded":"2024-03-12T03:14:17.000071+0000","last_fullsized":"2024-03-12T03:14:17.000071+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:28:59.215655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.616097+0000","last_change":"2024-03-12T03:13:42.937296+0000","last_active":"2024-03-12T03:14:16.616097+0000","last_peered":"2024-03-12T03:14:16.616097+0000","last_clean":"2024-03-12T03:14:16.616097+0000","last_became_active":"2024-03-12T03:13:42.937033+0000","last_became_peered":"2024-03-12T03:13:42.937033+0000","last_unstale":"2024-03-12T03:14:16.616097+0000","last_undegraded":"2024-03-12T03:14:16.616097+0000","last_fullsized":"2024-03-12T03:14:16.616097+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:39:04.707241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.508812+0000","last_change":"2024-03-12T03:13:44.997442+0000","last_active":"2024-03-12T03:14:16.508812+0000","last_peered":"2024-03-12T03:14:16.508812+0000","last_clean":"2024-03-12T03:14:16.508812+0000","last_became_active":"2024-03-12T03:13:44.997110+0000","last_became_peered":"2024-03-12T03:13:44.997110+0000","last_unstale":"2024-03-12T03:14:16.508812+0000","last_undegraded":"2024-03-12T03:14:16.508812+0000","last_fullsized":"2024-03-12T03:14:16.508812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:30:38.116899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510218+0000","last_change":"2024-03-12T03:13:38.917009+0000","last_active":"2024-03-12T03:14:16.510218+0000","last_peered":"2024-03-12T03:14:16.510218+0000","last_clean":"2024-03-12T03:14:16.510218+0000","last_became_active":"2024-03-12T03:13:38.916752+0000","last_became_peered":"2024-03-12T03:13:38.916752+0000","last_unstale":"2024-03-12T03:14:16.510218+0000","last_undegraded":"2024-03-12T03:14:16.510218+0000","last_fullsized":"2024-03-12T03:14:16.510218+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:59:37.556237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'25","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000862+0000","last_change":"2024-03-12T03:13:41.097510+0000","last_active":"2024-03-12T03:14:17.000862+0000","last_peered":"2024-03-12T03:14:17.000862+0000","last_clean":"2024-03-12T03:14:17.000862+0000","last_became_active":"2024-03-12T03:13:41.097205+0000","last_became_peered":"2024-03-12T03:13:41.097205+0000","last_unstale":"2024-03-12T03:14:17.000862+0000","last_undegraded":"2024-03-12T03:14:17.000862+0000","last_fullsized":"2024-03-12T03:14:17.000862+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:43:52.874518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999420+0000","last_change":"2024-03-12T03:13:42.936931+0000","last_active":"2024-03-12T03:14:16.999420+0000","last_peered":"2024-03-12T03:14:16.999420+0000","last_clean":"2024-03-12T03:14:16.999420+0000","last_became_active":"2024-03-12T03:13:42.936643+0000","last_became_peered":"2024-03-12T03:13:42.936643+0000","last_unstale":"2024-03-12T03:14:16.999420+0000","last_undegraded":"2024-03-12T03:14:16.999420+0000","last_fullsized":"2024-03-12T03:14:16.999420+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:57.012793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999577+0000","last_change":"2024-03-12T03:13:44.950852+0000","last_active":"2024-03-12T03:14:16.999577+0000","last_peered":"2024-03-12T03:14:16.999577+0000","last_clean":"2024-03-12T03:14:16.999577+0000","last_became_active":"2024-03-12T03:13:44.950501+0000","last_became_peered":"2024-03-12T03:13:44.950501+0000","last_unstale":"2024-03-12T03:14:16.999577+0000","last_undegraded":"2024-03-12T03:14:16.999577+0000","last_fullsized":"2024-03-12T03:14:16.999577+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:31:14.165775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615843+0000","last_change":"2024-03-12T03:13:38.925086+0000","last_active":"2024-03-12T03:14:16.615843+0000","last_peered":"2024-03-12T03:14:16.615843+0000","last_clean":"2024-03-12T03:14:16.615843+0000","last_became_active":"2024-03-12T03:13:38.924772+0000","last_became_peered":"2024-03-12T03:13:38.924772+0000","last_unstale":"2024-03-12T03:14:16.615843+0000","last_undegraded":"2024-03-12T03:14:16.615843+0000","last_fullsized":"2024-03-12T03:14:16.615843+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:51:26.322445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'16","reported_seq":56,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509465+0000","last_change":"2024-03-12T03:13:40.922975+0000","last_active":"2024-03-12T03:14:16.509465+0000","last_peered":"2024-03-12T03:14:16.509465+0000","last_clean":"2024-03-12T03:14:16.509465+0000","last_became_active":"2024-03-12T03:13:40.922569+0000","last_became_peered":"2024-03-12T03:13:40.922569+0000","last_unstale":"2024-03-12T03:14:16.509465+0000","last_undegraded":"2024-03-12T03:14:16.509465+0000","last_fullsized":"2024-03-12T03:14:16.509465+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:29:06.817123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.981736+0000","last_change":"2024-03-12T03:13:42.935058+0000","last_active":"2024-03-12T03:14:53.981736+0000","last_peered":"2024-03-12T03:14:53.981736+0000","last_clean":"2024-03-12T03:14:53.981736+0000","last_became_active":"2024-03-12T03:13:42.934725+0000","last_became_peered":"2024-03-12T03:13:42.934725+0000","last_unstale":"2024-03-12T03:14:53.981736+0000","last_undegraded":"2024-03-12T03:14:53.981736+0000","last_fullsized":"2024-03-12T03:14:53.981736+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:47:43.360515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627343+0000","last_change":"2024-03-12T03:13:44.997517+0000","last_active":"2024-03-12T03:14:16.627343+0000","last_peered":"2024-03-12T03:14:16.627343+0000","last_clean":"2024-03-12T03:14:16.627343+0000","last_became_active":"2024-03-12T03:13:44.997190+0000","last_became_peered":"2024-03-12T03:13:44.997190+0000","last_unstale":"2024-03-12T03:14:16.627343+0000","last_undegraded":"2024-03-12T03:14:16.627343+0000","last_fullsized":"2024-03-12T03:14:16.627343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:00:05.360461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000102+0000","last_change":"2024-03-12T03:13:38.919557+0000","last_active":"2024-03-12T03:14:17.000102+0000","last_peered":"2024-03-12T03:14:17.000102+0000","last_clean":"2024-03-12T03:14:17.000102+0000","last_became_active":"2024-03-12T03:13:38.919278+0000","last_became_peered":"2024-03-12T03:13:38.919278+0000","last_unstale":"2024-03-12T03:14:17.000102+0000","last_undegraded":"2024-03-12T03:14:17.000102+0000","last_fullsized":"2024-03-12T03:14:17.000102+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:03:51.500359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'20","reported_seq":62,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510177+0000","last_change":"2024-03-12T03:13:40.923874+0000","last_active":"2024-03-12T03:14:16.510177+0000","last_peered":"2024-03-12T03:14:16.510177+0000","last_clean":"2024-03-12T03:14:16.510177+0000","last_became_active":"2024-03-12T03:13:40.923540+0000","last_became_peered":"2024-03-12T03:13:40.923540+0000","last_unstale":"2024-03-12T03:14:16.510177+0000","last_undegraded":"2024-03-12T03:14:16.510177+0000","last_fullsized":"2024-03-12T03:14:16.510177+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:39:53.308593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.981735+0000","last_change":"2024-03-12T03:13:42.933668+0000","last_active":"2024-03-12T03:14:53.981735+0000","last_peered":"2024-03-12T03:14:53.981735+0000","last_clean":"2024-03-12T03:14:53.981735+0000","last_became_active":"2024-03-12T03:13:42.932841+0000","last_became_peered":"2024-03-12T03:13:42.932841+0000","last_unstale":"2024-03-12T03:14:53.981735+0000","last_undegraded":"2024-03-12T03:14:53.981735+0000","last_fullsized":"2024-03-12T03:14:53.981735+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:18:50.441610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510248+0000","last_change":"2024-03-12T03:13:44.947738+0000","last_active":"2024-03-12T03:14:16.510248+0000","last_peered":"2024-03-12T03:14:16.510248+0000","last_clean":"2024-03-12T03:14:16.510248+0000","last_became_active":"2024-03-12T03:13:44.947430+0000","last_became_peered":"2024-03-12T03:13:44.947430+0000","last_unstale":"2024-03-12T03:14:16.510248+0000","last_undegraded":"2024-03-12T03:14:16.510248+0000","last_fullsized":"2024-03-12T03:14:16.510248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:30:57.566324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.512111+0000","last_change":"2024-03-12T03:13:38.929141+0000","last_active":"2024-03-12T03:14:16.512111+0000","last_peered":"2024-03-12T03:14:16.512111+0000","last_clean":"2024-03-12T03:14:16.512111+0000","last_became_active":"2024-03-12T03:13:38.928839+0000","last_became_peered":"2024-03-12T03:13:38.928839+0000","last_unstale":"2024-03-12T03:14:16.512111+0000","last_undegraded":"2024-03-12T03:14:16.512111+0000","last_fullsized":"2024-03-12T03:14:16.512111+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:45.937151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'30","reported_seq":85,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509828+0000","last_change":"2024-03-12T03:13:41.097359+0000","last_active":"2024-03-12T03:14:16.509828+0000","last_peered":"2024-03-12T03:14:16.509828+0000","last_clean":"2024-03-12T03:14:16.509828+0000","last_became_active":"2024-03-12T03:13:41.097064+0000","last_became_peered":"2024-03-12T03:13:41.097064+0000","last_unstale":"2024-03-12T03:14:16.509828+0000","last_undegraded":"2024-03-12T03:14:16.509828+0000","last_fullsized":"2024-03-12T03:14:16.509828+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:25:25.853023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509939+0000","last_change":"2024-03-12T03:13:42.934233+0000","last_active":"2024-03-12T03:14:16.509939+0000","last_peered":"2024-03-12T03:14:16.509939+0000","last_clean":"2024-03-12T03:14:16.509939+0000","last_became_active":"2024-03-12T03:13:42.933939+0000","last_became_peered":"2024-03-12T03:13:42.933939+0000","last_unstale":"2024-03-12T03:14:16.509939+0000","last_undegraded":"2024-03-12T03:14:16.509939+0000","last_fullsized":"2024-03-12T03:14:16.509939+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:28.340329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615691+0000","last_change":"2024-03-12T03:13:44.942867+0000","last_active":"2024-03-12T03:14:16.615691+0000","last_peered":"2024-03-12T03:14:16.615691+0000","last_clean":"2024-03-12T03:14:16.615691+0000","last_became_active":"2024-03-12T03:13:44.942571+0000","last_became_peered":"2024-03-12T03:13:44.942571+0000","last_unstale":"2024-03-12T03:14:16.615691+0000","last_undegraded":"2024-03-12T03:14:16.615691+0000","last_fullsized":"2024-03-12T03:14:16.615691+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:24:41.603955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.982068+0000","last_change":"2024-03-12T03:13:42.932999+0000","last_active":"2024-03-12T03:14:53.982068+0000","last_peered":"2024-03-12T03:14:53.982068+0000","last_clean":"2024-03-12T03:14:53.982068+0000","last_became_active":"2024-03-12T03:13:42.932713+0000","last_became_peered":"2024-03-12T03:13:42.932713+0000","last_unstale":"2024-03-12T03:14:53.982068+0000","last_undegraded":"2024-03-12T03:14:53.982068+0000","last_fullsized":"2024-03-12T03:14:53.982068+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:43:32.261711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509740+0000","last_change":"2024-03-12T03:13:38.925544+0000","last_active":"2024-03-12T03:14:16.509740+0000","last_peered":"2024-03-12T03:14:16.509740+0000","last_clean":"2024-03-12T03:14:16.509740+0000","last_became_active":"2024-03-12T03:13:38.925258+0000","last_became_peered":"2024-03-12T03:13:38.925258+0000","last_unstale":"2024-03-12T03:14:16.509740+0000","last_undegraded":"2024-03-12T03:14:16.509740+0000","last_fullsized":"2024-03-12T03:14:16.509740+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:01:23.224690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.511481+0000","last_change":"2024-03-12T03:13:40.923543+0000","last_active":"2024-03-12T03:14:16.511481+0000","last_peered":"2024-03-12T03:14:16.511481+0000","last_clean":"2024-03-12T03:14:16.511481+0000","last_became_active":"2024-03-12T03:13:40.922942+0000","last_became_peered":"2024-03-12T03:13:40.922942+0000","last_unstale":"2024-03-12T03:14:16.511481+0000","last_undegraded":"2024-03-12T03:14:16.511481+0000","last_fullsized":"2024-03-12T03:14:16.511481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:45:42.789885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510681+0000","last_change":"2024-03-12T03:13:44.997399+0000","last_active":"2024-03-12T03:14:16.510681+0000","last_peered":"2024-03-12T03:14:16.510681+0000","last_clean":"2024-03-12T03:14:16.510681+0000","last_became_active":"2024-03-12T03:13:44.997070+0000","last_became_peered":"2024-03-12T03:13:44.997070+0000","last_unstale":"2024-03-12T03:14:16.510681+0000","last_undegraded":"2024-03-12T03:14:16.510681+0000","last_fullsized":"2024-03-12T03:14:16.510681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:28:03.546726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.616093+0000","last_change":"2024-03-12T03:13:42.939452+0000","last_active":"2024-03-12T03:14:16.616093+0000","last_peered":"2024-03-12T03:14:16.616093+0000","last_clean":"2024-03-12T03:14:16.616093+0000","last_became_active":"2024-03-12T03:13:42.939148+0000","last_became_peered":"2024-03-12T03:13:42.939148+0000","last_unstale":"2024-03-12T03:14:16.616093+0000","last_undegraded":"2024-03-12T03:14:16.616093+0000","last_fullsized":"2024-03-12T03:14:16.616093+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:19:14.944979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509945+0000","last_change":"2024-03-12T03:13:38.926088+0000","last_active":"2024-03-12T03:14:16.509945+0000","last_peered":"2024-03-12T03:14:16.509945+0000","last_clean":"2024-03-12T03:14:16.509945+0000","last_became_active":"2024-03-12T03:13:38.925616+0000","last_became_peered":"2024-03-12T03:13:38.925616+0000","last_unstale":"2024-03-12T03:14:16.509945+0000","last_undegraded":"2024-03-12T03:14:16.509945+0000","last_fullsized":"2024-03-12T03:14:16.509945+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:37:18.439932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'17","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.511064+0000","last_change":"2024-03-12T03:13:40.924337+0000","last_active":"2024-03-12T03:14:16.511064+0000","last_peered":"2024-03-12T03:14:16.511064+0000","last_clean":"2024-03-12T03:14:16.511064+0000","last_became_active":"2024-03-12T03:13:40.923975+0000","last_became_peered":"2024-03-12T03:13:40.923975+0000","last_unstale":"2024-03-12T03:14:16.511064+0000","last_undegraded":"2024-03-12T03:14:16.511064+0000","last_fullsized":"2024-03-12T03:14:16.511064+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:39:55.761682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.624756+0000","last_change":"2024-03-12T03:13:44.948402+0000","last_active":"2024-03-12T03:14:16.624756+0000","last_peered":"2024-03-12T03:14:16.624756+0000","last_clean":"2024-03-12T03:14:16.624756+0000","last_became_active":"2024-03-12T03:13:44.947990+0000","last_became_peered":"2024-03-12T03:13:44.947990+0000","last_unstale":"2024-03-12T03:14:16.624756+0000","last_undegraded":"2024-03-12T03:14:16.624756+0000","last_fullsized":"2024-03-12T03:14:16.624756+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:07:59.047651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510356+0000","last_change":"2024-03-12T03:13:42.941821+0000","last_active":"2024-03-12T03:14:16.510356+0000","last_peered":"2024-03-12T03:14:16.510356+0000","last_clean":"2024-03-12T03:14:16.510356+0000","last_became_active":"2024-03-12T03:13:42.941472+0000","last_became_peered":"2024-03-12T03:13:42.941472+0000","last_unstale":"2024-03-12T03:14:16.510356+0000","last_undegraded":"2024-03-12T03:14:16.510356+0000","last_fullsized":"2024-03-12T03:14:16.510356+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:02.076746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999302+0000","last_change":"2024-03-12T03:13:38.923139+0000","last_active":"2024-03-12T03:14:16.999302+0000","last_peered":"2024-03-12T03:14:16.999302+0000","last_clean":"2024-03-12T03:14:16.999302+0000","last_became_active":"2024-03-12T03:13:38.922772+0000","last_became_peered":"2024-03-12T03:13:38.922772+0000","last_unstale":"2024-03-12T03:14:16.999302+0000","last_undegraded":"2024-03-12T03:14:16.999302+0000","last_fullsized":"2024-03-12T03:14:16.999302+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:57.646908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510598+0000","last_change":"2024-03-12T03:13:40.927968+0000","last_active":"2024-03-12T03:14:16.510598+0000","last_peered":"2024-03-12T03:14:16.510598+0000","last_clean":"2024-03-12T03:14:16.510598+0000","last_became_active":"2024-03-12T03:13:40.927705+0000","last_became_peered":"2024-03-12T03:13:40.927705+0000","last_unstale":"2024-03-12T03:14:16.510598+0000","last_undegraded":"2024-03-12T03:14:16.510598+0000","last_fullsized":"2024-03-12T03:14:16.510598+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:53.556755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510974+0000","last_change":"2024-03-12T03:13:44.947632+0000","last_active":"2024-03-12T03:14:16.510974+0000","last_peered":"2024-03-12T03:14:16.510974+0000","last_clean":"2024-03-12T03:14:16.510974+0000","last_became_active":"2024-03-12T03:13:44.947325+0000","last_became_peered":"2024-03-12T03:13:44.947325+0000","last_unstale":"2024-03-12T03:14:16.510974+0000","last_undegraded":"2024-03-12T03:14:16.510974+0000","last_fullsized":"2024-03-12T03:14:16.510974+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:13:26.591561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.981741+0000","last_change":"2024-03-12T03:13:42.937388+0000","last_active":"2024-03-12T03:14:53.981741+0000","last_peered":"2024-03-12T03:14:53.981741+0000","last_clean":"2024-03-12T03:14:53.981741+0000","last_became_active":"2024-03-12T03:13:42.937030+0000","last_became_peered":"2024-03-12T03:13:42.937030+0000","last_unstale":"2024-03-12T03:14:53.981741+0000","last_undegraded":"2024-03-12T03:14:53.981741+0000","last_fullsized":"2024-03-12T03:14:53.981741+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:35:41.695226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510475+0000","last_change":"2024-03-12T03:13:38.926318+0000","last_active":"2024-03-12T03:14:16.510475+0000","last_peered":"2024-03-12T03:14:16.510475+0000","last_clean":"2024-03-12T03:14:16.510475+0000","last_became_active":"2024-03-12T03:13:38.925999+0000","last_became_peered":"2024-03-12T03:13:38.925999+0000","last_unstale":"2024-03-12T03:14:16.510475+0000","last_undegraded":"2024-03-12T03:14:16.510475+0000","last_fullsized":"2024-03-12T03:14:16.510475+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:11:46.388826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510899+0000","last_change":"2024-03-12T03:13:40.924917+0000","last_active":"2024-03-12T03:14:16.510899+0000","last_peered":"2024-03-12T03:14:16.510899+0000","last_clean":"2024-03-12T03:14:16.510899+0000","last_became_active":"2024-03-12T03:13:40.924581+0000","last_became_peered":"2024-03-12T03:13:40.924581+0000","last_unstale":"2024-03-12T03:14:16.510899+0000","last_undegraded":"2024-03-12T03:14:16.510899+0000","last_fullsized":"2024-03-12T03:14:16.510899+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:25.698015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":587,"ondisk_log_size":587,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":72,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1314816,"data_stored":872356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712700,"statfs":{"total":95995035648,"available":95961804800,"internally_reserved":0,"allocated":1302528,"data_stored":872651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717408,"statfs":{"total":95995035648,"available":95966625792,"internally_reserved":0,"allocated":741376,"data_stored":282158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32468,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712684,"statfs":{"total":95995035648,"available":95961788416,"internally_reserved":0,"allocated":1323008,"data_stored":873432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":712704,"data_stored":281491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":281814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":724992,"data_stored":281528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":712704,"data_stored":282706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:15:04.496 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:15:04.744 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:15:04.768 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:04 smithi094 ceph-mon[40484]: from='client.14757 v1:172.21.15.94:0/1503368215' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:15:04.768 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:04 smithi094 ceph-mon[40484]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:04.768 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:04 smithi094 ceph-mon[29291]: from='client.14757 v1:172.21.15.94:0/1503368215' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:15:04.768 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:04 smithi094 ceph-mon[29291]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:04 smithi151 ceph-mon[32923]: from='client.14757 v1:172.21.15.94:0/1503368215' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:15:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:04 smithi151 ceph-mon[32923]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:06.690 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:06 smithi094 ceph-mon[40484]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:06.691 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:06 smithi094 ceph-mon[29291]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:06.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.687+0000 7f3421b28700 1 -- 172.21.15.94:0/2817167631 >> v1:172.21.15.94:6789/0 conn(0x7f341c10d520 legacy=0x7f341c10f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:06.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.687+0000 7f3421b28700 1 -- 172.21.15.94:0/2817167631 shutdown_connections 2024-03-12T03:15:06.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.687+0000 7f3421b28700 1 -- 172.21.15.94:0/2817167631 >> 172.21.15.94:0/2817167631 conn(0x7f341c0fc5b0 msgr2=0x7f341c0fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:06.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.687+0000 7f3421b28700 1 -- 172.21.15.94:0/2817167631 shutdown_connections 2024-03-12T03:15:06.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.688+0000 7f3421b28700 1 -- 172.21.15.94:0/2817167631 wait complete. 2024-03-12T03:15:06.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.688+0000 7f3421b28700 1 Processor -- start 2024-03-12T03:15:06.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.688+0000 7f3421b28700 1 -- start start 2024-03-12T03:15:06.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3421b28700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f341c1a5b00 con 0x7f341c10d520 2024-03-12T03:15:06.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3421b28700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f341c1a6d20 con 0x7f341c109950 2024-03-12T03:15:06.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3421b28700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f341c1a7f60 con 0x7f341c106aa0 2024-03-12T03:15:06.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f341b7fe700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f341c106aa0 0x7f341c10cc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:53804/0 (socket says 172.21.15.94:53804) 2024-03-12T03:15:06.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f341b7fe700 1 -- 172.21.15.94:0/450834611 learned_addr learned my addr 172.21.15.94:0/450834611 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:15:06.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067461118 0 0) 0x7f341c1a7f60 con 0x7f341c106aa0 2024-03-12T03:15:06.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34040031f0 con 0x7f341c106aa0 2024-03-12T03:15:06.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973248741 0 0) 0x7f341c1a5b00 con 0x7f341c10d520 2024-03-12T03:15:06.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f341c1a7f60 con 0x7f341c10d520 2024-03-12T03:15:06.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188184369 0 0) 0x7f341c1a6d20 con 0x7f341c109950 2024-03-12T03:15:06.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f341c1a5b00 con 0x7f341c109950 2024-03-12T03:15:06.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 619261642 0 0) 0x7f341c1a7f60 con 0x7f341c10d520 2024-03-12T03:15:06.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f341c1a6d20 con 0x7f341c10d520 2024-03-12T03:15:06.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f3414001d20 con 0x7f341c10d520 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2387591363 0 0) 0x7f34040031f0 con 0x7f341c106aa0 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f341c1a7f60 con 0x7f341c106aa0 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.689+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3242099116 0 0) 0x7f341c1a5b00 con 0x7f341c109950 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.690+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34040031f0 con 0x7f341c109950 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.690+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f340c0018b0 con 0x7f341c106aa0 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.690+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f3410001840 con 0x7f341c109950 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.690+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3605750763 0 0) 0x7f341c1a6d20 con 0x7f341c10d520 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.690+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 >> v1:172.21.15.94:6790/0 conn(0x7f341c106aa0 legacy=0x7f341c10cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.690+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 >> v1:172.21.15.151:6789/0 conn(0x7f341c109950 legacy=0x7f341c1a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.690+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c1a91a0 con 0x7f341c10d520 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.690+0000 7f3421b28700 1 -- 172.21.15.94:0/450834611 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341c1a5d30 con 0x7f341c10d520 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.690+0000 7f3421b28700 1 -- 172.21.15.94:0/450834611 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f341c1a61f0 con 0x7f341c10d520 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.690+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f3414002010 con 0x7f341c10d520 2024-03-12T03:15:06.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.690+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f3414004f50 con 0x7f341c10d520 2024-03-12T03:15:06.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.691+0000 7f3421b28700 1 -- 172.21.15.94:0/450834611 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3408005150 con 0x7f341c10d520 2024-03-12T03:15:06.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.691+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f3414003a90 con 0x7f341c10d520 2024-03-12T03:15:06.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.692+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7f34140882a0 con 0x7f341c10d520 2024-03-12T03:15:06.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.696+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f3414055ae0 con 0x7f341c10d520 2024-03-12T03:15:06.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.887+0000 7f3421b28700 1 -- 172.21.15.94:0/450834611 --> v1:172.21.15.94:6800/2099808905 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3408000bc0 con 0x7f340406ee30 2024-03-12T03:15:06.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.896+0000 7f3418ff9700 1 -- 172.21.15.94:0/450834611 <== mgr.34490 v1:172.21.15.94:6800/2099808905 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346507 (unknown 2965378022 0 2353166569) 0x7f3408000bc0 con 0x7f340406ee30 2024-03-12T03:15:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:15:06.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.899+0000 7f3421b28700 1 -- 172.21.15.94:0/450834611 >> v1:172.21.15.94:6800/2099808905 conn(0x7f340406ee30 legacy=0x7f34040712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:06.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.899+0000 7f3421b28700 1 -- 172.21.15.94:0/450834611 >> v1:172.21.15.94:6789/0 conn(0x7f341c10d520 legacy=0x7f341c1a41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:06.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.899+0000 7f3421b28700 1 -- 172.21.15.94:0/450834611 shutdown_connections 2024-03-12T03:15:06.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.899+0000 7f3421b28700 1 -- 172.21.15.94:0/450834611 >> 172.21.15.94:0/450834611 conn(0x7f341c0fc5b0 msgr2=0x7f341c100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:06.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.899+0000 7f3421b28700 1 -- 172.21.15.94:0/450834611 shutdown_connections 2024-03-12T03:15:06.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:06.899+0000 7f3421b28700 1 -- 172.21.15.94:0/450834611 wait complete. 2024-03-12T03:15:06.906 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-12T03:15:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:06 smithi151 ceph-mon[32923]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:07.683 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-03-12T03:15:06.529984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1299,"num_read_kb":1113,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":231864,"kb_used_data":7388,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749729352,"statfs":{"total":767960285184,"available":767722856448,"internally_reserved":0,"allocated":7565312,"data_stored":4028136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":229756465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002991"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510147+0000","last_change":"2024-03-12T03:13:44.944304+0000","last_active":"2024-03-12T03:14:16.510147+0000","last_peered":"2024-03-12T03:14:16.510147+0000","last_clean":"2024-03-12T03:14:16.510147+0000","last_became_active":"2024-03-12T03:13:44.943975+0000","last_became_peered":"2024-03-12T03:13:44.943975+0000","last_unstale":"2024-03-12T03:14:16.510147+0000","last_undegraded":"2024-03-12T03:14:16.510147+0000","last_fullsized":"2024-03-12T03:14:16.510147+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:05:41.838808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.508913+0000","last_change":"2024-03-12T03:13:42.940899+0000","last_active":"2024-03-12T03:14:16.508913+0000","last_peered":"2024-03-12T03:14:16.508913+0000","last_clean":"2024-03-12T03:14:16.508913+0000","last_became_active":"2024-03-12T03:13:42.940414+0000","last_became_peered":"2024-03-12T03:13:42.940414+0000","last_unstale":"2024-03-12T03:14:16.508913+0000","last_undegraded":"2024-03-12T03:14:16.508913+0000","last_fullsized":"2024-03-12T03:14:16.508913+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:52:38.732745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999795+0000","last_change":"2024-03-12T03:13:38.913213+0000","last_active":"2024-03-12T03:14:16.999795+0000","last_peered":"2024-03-12T03:14:16.999795+0000","last_clean":"2024-03-12T03:14:16.999795+0000","last_became_active":"2024-03-12T03:13:38.912803+0000","last_became_peered":"2024-03-12T03:13:38.912803+0000","last_unstale":"2024-03-12T03:14:16.999795+0000","last_undegraded":"2024-03-12T03:14:16.999795+0000","last_fullsized":"2024-03-12T03:14:16.999795+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:49:25.201096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'15","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509886+0000","last_change":"2024-03-12T03:13:40.924525+0000","last_active":"2024-03-12T03:14:16.509886+0000","last_peered":"2024-03-12T03:14:16.509886+0000","last_clean":"2024-03-12T03:14:16.509886+0000","last_became_active":"2024-03-12T03:13:40.923702+0000","last_became_peered":"2024-03-12T03:13:40.923702+0000","last_unstale":"2024-03-12T03:14:16.509886+0000","last_undegraded":"2024-03-12T03:14:16.509886+0000","last_fullsized":"2024-03-12T03:14:16.509886+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:45:47.871678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510317+0000","last_change":"2024-03-12T03:13:44.999070+0000","last_active":"2024-03-12T03:14:16.510317+0000","last_peered":"2024-03-12T03:14:16.510317+0000","last_clean":"2024-03-12T03:14:16.510317+0000","last_became_active":"2024-03-12T03:13:44.998667+0000","last_became_peered":"2024-03-12T03:13:44.998667+0000","last_unstale":"2024-03-12T03:14:16.510317+0000","last_undegraded":"2024-03-12T03:14:16.510317+0000","last_fullsized":"2024-03-12T03:14:16.510317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:48:01.866371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509889+0000","last_change":"2024-03-12T03:13:42.937335+0000","last_active":"2024-03-12T03:14:16.509889+0000","last_peered":"2024-03-12T03:14:16.509889+0000","last_clean":"2024-03-12T03:14:16.509889+0000","last_became_active":"2024-03-12T03:13:42.934402+0000","last_became_peered":"2024-03-12T03:13:42.934402+0000","last_unstale":"2024-03-12T03:14:16.509889+0000","last_undegraded":"2024-03-12T03:14:16.509889+0000","last_fullsized":"2024-03-12T03:14:16.509889+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:21:24.114718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.511864+0000","last_change":"2024-03-12T03:13:38.924780+0000","last_active":"2024-03-12T03:14:16.511864+0000","last_peered":"2024-03-12T03:14:16.511864+0000","last_clean":"2024-03-12T03:14:16.511864+0000","last_became_active":"2024-03-12T03:13:38.924533+0000","last_became_peered":"2024-03-12T03:13:38.924533+0000","last_unstale":"2024-03-12T03:14:16.511864+0000","last_undegraded":"2024-03-12T03:14:16.511864+0000","last_fullsized":"2024-03-12T03:14:16.511864+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:30.316175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'39","reported_seq":102,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.513216+0000","last_change":"2024-03-12T03:13:40.928759+0000","last_active":"2024-03-12T03:14:16.513216+0000","last_peered":"2024-03-12T03:14:16.513216+0000","last_clean":"2024-03-12T03:14:16.513216+0000","last_became_active":"2024-03-12T03:13:40.928473+0000","last_became_peered":"2024-03-12T03:13:40.928473+0000","last_unstale":"2024-03-12T03:14:16.513216+0000","last_undegraded":"2024-03-12T03:14:16.513216+0000","last_fullsized":"2024-03-12T03:14:16.513216+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:00:02.208063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.998591+0000","last_change":"2024-03-12T03:13:44.945990+0000","last_active":"2024-03-12T03:14:16.998591+0000","last_peered":"2024-03-12T03:14:16.998591+0000","last_clean":"2024-03-12T03:14:16.998591+0000","last_became_active":"2024-03-12T03:13:44.945653+0000","last_became_peered":"2024-03-12T03:13:44.945653+0000","last_unstale":"2024-03-12T03:14:16.998591+0000","last_undegraded":"2024-03-12T03:14:16.998591+0000","last_fullsized":"2024-03-12T03:14:16.998591+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:50:14.933362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615558+0000","last_change":"2024-03-12T03:13:42.941836+0000","last_active":"2024-03-12T03:14:16.615558+0000","last_peered":"2024-03-12T03:14:16.615558+0000","last_clean":"2024-03-12T03:14:16.615558+0000","last_became_active":"2024-03-12T03:13:42.940973+0000","last_became_peered":"2024-03-12T03:13:42.940973+0000","last_unstale":"2024-03-12T03:14:16.615558+0000","last_undegraded":"2024-03-12T03:14:16.615558+0000","last_fullsized":"2024-03-12T03:14:16.615558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:39:55.057516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615737+0000","last_change":"2024-03-12T03:13:38.919541+0000","last_active":"2024-03-12T03:14:16.615737+0000","last_peered":"2024-03-12T03:14:16.615737+0000","last_clean":"2024-03-12T03:14:16.615737+0000","last_became_active":"2024-03-12T03:13:38.919242+0000","last_became_peered":"2024-03-12T03:13:38.919242+0000","last_unstale":"2024-03-12T03:14:16.615737+0000","last_undegraded":"2024-03-12T03:14:16.615737+0000","last_fullsized":"2024-03-12T03:14:16.615737+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:50:27.245276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'21","reported_seq":69,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510393+0000","last_change":"2024-03-12T03:13:40.926045+0000","last_active":"2024-03-12T03:14:16.510393+0000","last_peered":"2024-03-12T03:14:16.510393+0000","last_clean":"2024-03-12T03:14:16.510393+0000","last_became_active":"2024-03-12T03:13:40.925733+0000","last_became_peered":"2024-03-12T03:13:40.925733+0000","last_unstale":"2024-03-12T03:14:16.510393+0000","last_undegraded":"2024-03-12T03:14:16.510393+0000","last_fullsized":"2024-03-12T03:14:16.510393+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:18:36.134706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615317+0000","last_change":"2024-03-12T03:13:44.940804+0000","last_active":"2024-03-12T03:14:16.615317+0000","last_peered":"2024-03-12T03:14:16.615317+0000","last_clean":"2024-03-12T03:14:16.615317+0000","last_became_active":"2024-03-12T03:13:44.939677+0000","last_became_peered":"2024-03-12T03:13:44.939677+0000","last_unstale":"2024-03-12T03:14:16.615317+0000","last_undegraded":"2024-03-12T03:14:16.615317+0000","last_fullsized":"2024-03-12T03:14:16.615317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:06:06.856376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.626990+0000","last_change":"2024-03-12T03:13:42.941547+0000","last_active":"2024-03-12T03:14:16.626990+0000","last_peered":"2024-03-12T03:14:16.626990+0000","last_clean":"2024-03-12T03:14:16.626990+0000","last_became_active":"2024-03-12T03:13:42.941131+0000","last_became_peered":"2024-03-12T03:13:42.941131+0000","last_unstale":"2024-03-12T03:14:16.626990+0000","last_undegraded":"2024-03-12T03:14:16.626990+0000","last_fullsized":"2024-03-12T03:14:16.626990+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:16:33.264911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615610+0000","last_change":"2024-03-12T03:13:38.921507+0000","last_active":"2024-03-12T03:14:16.615610+0000","last_peered":"2024-03-12T03:14:16.615610+0000","last_clean":"2024-03-12T03:14:16.615610+0000","last_became_active":"2024-03-12T03:13:38.921229+0000","last_became_peered":"2024-03-12T03:13:38.921229+0000","last_unstale":"2024-03-12T03:14:16.615610+0000","last_undegraded":"2024-03-12T03:14:16.615610+0000","last_fullsized":"2024-03-12T03:14:16.615610+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:05:06.491743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'11","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510287+0000","last_change":"2024-03-12T03:13:40.922981+0000","last_active":"2024-03-12T03:14:16.510287+0000","last_peered":"2024-03-12T03:14:16.510287+0000","last_clean":"2024-03-12T03:14:16.510287+0000","last_became_active":"2024-03-12T03:13:40.922478+0000","last_became_peered":"2024-03-12T03:13:40.922478+0000","last_unstale":"2024-03-12T03:14:16.510287+0000","last_undegraded":"2024-03-12T03:14:16.510287+0000","last_fullsized":"2024-03-12T03:14:16.510287+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:08.463482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510795+0000","last_change":"2024-03-12T03:13:44.997335+0000","last_active":"2024-03-12T03:14:16.510795+0000","last_peered":"2024-03-12T03:14:16.510795+0000","last_clean":"2024-03-12T03:14:16.510795+0000","last_became_active":"2024-03-12T03:13:44.996990+0000","last_became_peered":"2024-03-12T03:13:44.996990+0000","last_unstale":"2024-03-12T03:14:16.510795+0000","last_undegraded":"2024-03-12T03:14:16.510795+0000","last_fullsized":"2024-03-12T03:14:16.510795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:45:35.639293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510785+0000","last_change":"2024-03-12T03:13:42.940699+0000","last_active":"2024-03-12T03:14:16.510785+0000","last_peered":"2024-03-12T03:14:16.510785+0000","last_clean":"2024-03-12T03:14:16.510785+0000","last_became_active":"2024-03-12T03:13:42.940381+0000","last_became_peered":"2024-03-12T03:13:42.940381+0000","last_unstale":"2024-03-12T03:14:16.510785+0000","last_undegraded":"2024-03-12T03:14:16.510785+0000","last_fullsized":"2024-03-12T03:14:16.510785+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:42:11.488043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000775+0000","last_change":"2024-03-12T03:13:38.925929+0000","last_active":"2024-03-12T03:14:17.000775+0000","last_peered":"2024-03-12T03:14:17.000775+0000","last_clean":"2024-03-12T03:14:17.000775+0000","last_became_active":"2024-03-12T03:13:38.925023+0000","last_became_peered":"2024-03-12T03:13:38.925023+0000","last_unstale":"2024-03-12T03:14:17.000775+0000","last_undegraded":"2024-03-12T03:14:17.000775+0000","last_fullsized":"2024-03-12T03:14:17.000775+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:23.058114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.624291+0000","last_change":"2024-03-12T03:13:40.925810+0000","last_active":"2024-03-12T03:14:16.624291+0000","last_peered":"2024-03-12T03:14:16.624291+0000","last_clean":"2024-03-12T03:14:16.624291+0000","last_became_active":"2024-03-12T03:13:40.925512+0000","last_became_peered":"2024-03-12T03:13:40.925512+0000","last_unstale":"2024-03-12T03:14:16.624291+0000","last_undegraded":"2024-03-12T03:14:16.624291+0000","last_fullsized":"2024-03-12T03:14:16.624291+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:31:00.134618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.512434+0000","last_change":"2024-03-12T03:13:44.997361+0000","last_active":"2024-03-12T03:14:16.512434+0000","last_peered":"2024-03-12T03:14:16.512434+0000","last_clean":"2024-03-12T03:14:16.512434+0000","last_became_active":"2024-03-12T03:13:44.997016+0000","last_became_peered":"2024-03-12T03:13:44.997016+0000","last_unstale":"2024-03-12T03:14:16.512434+0000","last_undegraded":"2024-03-12T03:14:16.512434+0000","last_fullsized":"2024-03-12T03:14:16.512434+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:40:03.641863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510815+0000","last_change":"2024-03-12T03:13:42.940153+0000","last_active":"2024-03-12T03:14:16.510815+0000","last_peered":"2024-03-12T03:14:16.510815+0000","last_clean":"2024-03-12T03:14:16.510815+0000","last_became_active":"2024-03-12T03:13:42.939411+0000","last_became_peered":"2024-03-12T03:13:42.939411+0000","last_unstale":"2024-03-12T03:14:16.510815+0000","last_undegraded":"2024-03-12T03:14:16.510815+0000","last_fullsized":"2024-03-12T03:14:16.510815+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:55:41.812233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510532+0000","last_change":"2024-03-12T03:13:38.921057+0000","last_active":"2024-03-12T03:14:16.510532+0000","last_peered":"2024-03-12T03:14:16.510532+0000","last_clean":"2024-03-12T03:14:16.510532+0000","last_became_active":"2024-03-12T03:13:38.920757+0000","last_became_peered":"2024-03-12T03:13:38.920757+0000","last_unstale":"2024-03-12T03:14:16.510532+0000","last_undegraded":"2024-03-12T03:14:16.510532+0000","last_fullsized":"2024-03-12T03:14:16.510532+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:25:50.682120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'18","reported_seq":84,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.513044+0000","last_change":"2024-03-12T03:14:04.012274+0000","last_active":"2024-03-12T03:14:16.513044+0000","last_peered":"2024-03-12T03:14:16.513044+0000","last_clean":"2024-03-12T03:14:16.513044+0000","last_became_active":"2024-03-12T03:14:04.005636+0000","last_became_peered":"2024-03-12T03:14:04.005636+0000","last_unstale":"2024-03-12T03:14:16.513044+0000","last_undegraded":"2024-03-12T03:14:16.513044+0000","last_fullsized":"2024-03-12T03:14:16.513044+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:16:16.943541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627359+0000","last_change":"2024-03-12T03:13:44.949139+0000","last_active":"2024-03-12T03:14:16.627359+0000","last_peered":"2024-03-12T03:14:16.627359+0000","last_clean":"2024-03-12T03:14:16.627359+0000","last_became_active":"2024-03-12T03:13:44.948626+0000","last_became_peered":"2024-03-12T03:13:44.948626+0000","last_unstale":"2024-03-12T03:14:16.627359+0000","last_undegraded":"2024-03-12T03:14:16.627359+0000","last_fullsized":"2024-03-12T03:14:16.627359+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:35:45.981328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.982068+0000","last_change":"2024-03-12T03:13:42.944817+0000","last_active":"2024-03-12T03:14:53.982068+0000","last_peered":"2024-03-12T03:14:53.982068+0000","last_clean":"2024-03-12T03:14:53.982068+0000","last_became_active":"2024-03-12T03:13:42.944163+0000","last_became_peered":"2024-03-12T03:13:42.944163+0000","last_unstale":"2024-03-12T03:14:53.982068+0000","last_undegraded":"2024-03-12T03:14:53.982068+0000","last_fullsized":"2024-03-12T03:14:53.982068+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:07:51.947025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509005+0000","last_change":"2024-03-12T03:13:38.921827+0000","last_active":"2024-03-12T03:14:16.509005+0000","last_peered":"2024-03-12T03:14:16.509005+0000","last_clean":"2024-03-12T03:14:16.509005+0000","last_became_active":"2024-03-12T03:13:38.921535+0000","last_became_peered":"2024-03-12T03:13:38.921535+0000","last_unstale":"2024-03-12T03:14:16.509005+0000","last_undegraded":"2024-03-12T03:14:16.509005+0000","last_fullsized":"2024-03-12T03:14:16.509005+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:13:46.740854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'16","reported_seq":56,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000224+0000","last_change":"2024-03-12T03:13:41.098610+0000","last_active":"2024-03-12T03:14:17.000224+0000","last_peered":"2024-03-12T03:14:17.000224+0000","last_clean":"2024-03-12T03:14:17.000224+0000","last_became_active":"2024-03-12T03:13:41.098331+0000","last_became_peered":"2024-03-12T03:13:41.098331+0000","last_unstale":"2024-03-12T03:14:17.000224+0000","last_undegraded":"2024-03-12T03:14:17.000224+0000","last_fullsized":"2024-03-12T03:14:17.000224+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:48:16.277359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.508721+0000","last_change":"2024-03-12T03:13:44.942886+0000","last_active":"2024-03-12T03:14:16.508721+0000","last_peered":"2024-03-12T03:14:16.508721+0000","last_clean":"2024-03-12T03:14:16.508721+0000","last_became_active":"2024-03-12T03:13:44.942558+0000","last_became_peered":"2024-03-12T03:13:44.942558+0000","last_unstale":"2024-03-12T03:14:16.508721+0000","last_undegraded":"2024-03-12T03:14:16.508721+0000","last_fullsized":"2024-03-12T03:14:16.508721+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:34:51.142099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000370+0000","last_change":"2024-03-12T03:13:42.936686+0000","last_active":"2024-03-12T03:14:17.000370+0000","last_peered":"2024-03-12T03:14:17.000370+0000","last_clean":"2024-03-12T03:14:17.000370+0000","last_became_active":"2024-03-12T03:13:42.935514+0000","last_became_peered":"2024-03-12T03:13:42.935514+0000","last_unstale":"2024-03-12T03:14:17.000370+0000","last_undegraded":"2024-03-12T03:14:17.000370+0000","last_fullsized":"2024-03-12T03:14:17.000370+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:15:03.926327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509528+0000","last_change":"2024-03-12T03:13:38.927276+0000","last_active":"2024-03-12T03:14:16.509528+0000","last_peered":"2024-03-12T03:14:16.509528+0000","last_clean":"2024-03-12T03:14:16.509528+0000","last_became_active":"2024-03-12T03:13:38.926916+0000","last_became_peered":"2024-03-12T03:13:38.926916+0000","last_unstale":"2024-03-12T03:14:16.509528+0000","last_undegraded":"2024-03-12T03:14:16.509528+0000","last_fullsized":"2024-03-12T03:14:16.509528+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:19:14.943793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'17","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999850+0000","last_change":"2024-03-12T03:13:40.929310+0000","last_active":"2024-03-12T03:14:16.999850+0000","last_peered":"2024-03-12T03:14:16.999850+0000","last_clean":"2024-03-12T03:14:16.999850+0000","last_became_active":"2024-03-12T03:13:40.928990+0000","last_became_peered":"2024-03-12T03:13:40.928990+0000","last_unstale":"2024-03-12T03:14:16.999850+0000","last_undegraded":"2024-03-12T03:14:16.999850+0000","last_fullsized":"2024-03-12T03:14:16.999850+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:33:36.214911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.626996+0000","last_change":"2024-03-12T03:13:44.946379+0000","last_active":"2024-03-12T03:14:16.626996+0000","last_peered":"2024-03-12T03:14:16.626996+0000","last_clean":"2024-03-12T03:14:16.626996+0000","last_became_active":"2024-03-12T03:13:44.946095+0000","last_became_peered":"2024-03-12T03:13:44.946095+0000","last_unstale":"2024-03-12T03:14:16.626996+0000","last_undegraded":"2024-03-12T03:14:16.626996+0000","last_fullsized":"2024-03-12T03:14:16.626996+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:43:08.958607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.998743+0000","last_change":"2024-03-12T03:13:42.942423+0000","last_active":"2024-03-12T03:14:16.998743+0000","last_peered":"2024-03-12T03:14:16.998743+0000","last_clean":"2024-03-12T03:14:16.998743+0000","last_became_active":"2024-03-12T03:13:42.942038+0000","last_became_peered":"2024-03-12T03:13:42.942038+0000","last_unstale":"2024-03-12T03:14:16.998743+0000","last_undegraded":"2024-03-12T03:14:16.998743+0000","last_fullsized":"2024-03-12T03:14:16.998743+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:41:46.231848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999873+0000","last_change":"2024-03-12T03:13:38.923824+0000","last_active":"2024-03-12T03:14:16.999873+0000","last_peered":"2024-03-12T03:14:16.999873+0000","last_clean":"2024-03-12T03:14:16.999873+0000","last_became_active":"2024-03-12T03:13:38.923440+0000","last_became_peered":"2024-03-12T03:13:38.923440+0000","last_unstale":"2024-03-12T03:14:16.999873+0000","last_undegraded":"2024-03-12T03:14:16.999873+0000","last_fullsized":"2024-03-12T03:14:16.999873+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:22:06.378441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":35,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509630+0000","last_change":"2024-03-12T03:13:40.923766+0000","last_active":"2024-03-12T03:14:16.509630+0000","last_peered":"2024-03-12T03:14:16.509630+0000","last_clean":"2024-03-12T03:14:16.509630+0000","last_became_active":"2024-03-12T03:13:40.923460+0000","last_became_peered":"2024-03-12T03:13:40.923460+0000","last_unstale":"2024-03-12T03:14:16.509630+0000","last_undegraded":"2024-03-12T03:14:16.509630+0000","last_fullsized":"2024-03-12T03:14:16.509630+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:37:06.539360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510145+0000","last_change":"2024-03-12T03:13:44.998732+0000","last_active":"2024-03-12T03:14:16.510145+0000","last_peered":"2024-03-12T03:14:16.510145+0000","last_clean":"2024-03-12T03:14:16.510145+0000","last_became_active":"2024-03-12T03:13:44.997545+0000","last_became_peered":"2024-03-12T03:13:44.997545+0000","last_unstale":"2024-03-12T03:14:16.510145+0000","last_undegraded":"2024-03-12T03:14:16.510145+0000","last_fullsized":"2024-03-12T03:14:16.510145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:27:31.542973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627165+0000","last_change":"2024-03-12T03:13:42.944935+0000","last_active":"2024-03-12T03:14:16.627165+0000","last_peered":"2024-03-12T03:14:16.627165+0000","last_clean":"2024-03-12T03:14:16.627165+0000","last_became_active":"2024-03-12T03:13:42.944626+0000","last_became_peered":"2024-03-12T03:13:42.944626+0000","last_unstale":"2024-03-12T03:14:16.627165+0000","last_undegraded":"2024-03-12T03:14:16.627165+0000","last_fullsized":"2024-03-12T03:14:16.627165+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:31:46.968518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615884+0000","last_change":"2024-03-12T03:13:38.920375+0000","last_active":"2024-03-12T03:14:16.615884+0000","last_peered":"2024-03-12T03:14:16.615884+0000","last_clean":"2024-03-12T03:14:16.615884+0000","last_became_active":"2024-03-12T03:13:38.920074+0000","last_became_peered":"2024-03-12T03:13:38.920074+0000","last_unstale":"2024-03-12T03:14:16.615884+0000","last_undegraded":"2024-03-12T03:14:16.615884+0000","last_fullsized":"2024-03-12T03:14:16.615884+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:23:34.980491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'17","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.513207+0000","last_change":"2024-03-12T03:13:41.097565+0000","last_active":"2024-03-12T03:14:16.513207+0000","last_peered":"2024-03-12T03:14:16.513207+0000","last_clean":"2024-03-12T03:14:16.513207+0000","last_became_active":"2024-03-12T03:13:41.097163+0000","last_became_peered":"2024-03-12T03:13:41.097163+0000","last_unstale":"2024-03-12T03:14:16.513207+0000","last_undegraded":"2024-03-12T03:14:16.513207+0000","last_fullsized":"2024-03-12T03:14:16.513207+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:36.405442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999867+0000","last_change":"2024-03-12T03:13:44.947369+0000","last_active":"2024-03-12T03:14:16.999867+0000","last_peered":"2024-03-12T03:14:16.999867+0000","last_clean":"2024-03-12T03:14:16.999867+0000","last_became_active":"2024-03-12T03:13:44.947062+0000","last_became_peered":"2024-03-12T03:13:44.947062+0000","last_unstale":"2024-03-12T03:14:16.999867+0000","last_undegraded":"2024-03-12T03:14:16.999867+0000","last_fullsized":"2024-03-12T03:14:16.999867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:15:30.842885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510399+0000","last_change":"2024-03-12T03:13:42.939145+0000","last_active":"2024-03-12T03:14:16.510399+0000","last_peered":"2024-03-12T03:14:16.510399+0000","last_clean":"2024-03-12T03:14:16.510399+0000","last_became_active":"2024-03-12T03:13:42.938795+0000","last_became_peered":"2024-03-12T03:13:42.938795+0000","last_unstale":"2024-03-12T03:14:16.510399+0000","last_undegraded":"2024-03-12T03:14:16.510399+0000","last_fullsized":"2024-03-12T03:14:16.510399+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:09:01.041567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627374+0000","last_change":"2024-03-12T03:13:38.924544+0000","last_active":"2024-03-12T03:14:16.627374+0000","last_peered":"2024-03-12T03:14:16.627374+0000","last_clean":"2024-03-12T03:14:16.627374+0000","last_became_active":"2024-03-12T03:13:38.924274+0000","last_became_peered":"2024-03-12T03:13:38.924274+0000","last_unstale":"2024-03-12T03:14:16.627374+0000","last_undegraded":"2024-03-12T03:14:16.627374+0000","last_fullsized":"2024-03-12T03:14:16.627374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:43:37.403267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509577+0000","last_change":"2024-03-12T03:13:40.926321+0000","last_active":"2024-03-12T03:14:16.509577+0000","last_peered":"2024-03-12T03:14:16.509577+0000","last_clean":"2024-03-12T03:14:16.509577+0000","last_became_active":"2024-03-12T03:13:40.924605+0000","last_became_peered":"2024-03-12T03:13:40.924605+0000","last_unstale":"2024-03-12T03:14:16.509577+0000","last_undegraded":"2024-03-12T03:14:16.509577+0000","last_fullsized":"2024-03-12T03:14:16.509577+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:55:43.295784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.512939+0000","last_change":"2024-03-12T03:13:44.946755+0000","last_active":"2024-03-12T03:14:16.512939+0000","last_peered":"2024-03-12T03:14:16.512939+0000","last_clean":"2024-03-12T03:14:16.512939+0000","last_became_active":"2024-03-12T03:13:44.946404+0000","last_became_peered":"2024-03-12T03:13:44.946404+0000","last_unstale":"2024-03-12T03:14:16.512939+0000","last_undegraded":"2024-03-12T03:14:16.512939+0000","last_fullsized":"2024-03-12T03:14:16.512939+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:47:34.387479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509390+0000","last_change":"2024-03-12T03:13:42.935816+0000","last_active":"2024-03-12T03:14:16.509390+0000","last_peered":"2024-03-12T03:14:16.509390+0000","last_clean":"2024-03-12T03:14:16.509390+0000","last_became_active":"2024-03-12T03:13:42.935482+0000","last_became_peered":"2024-03-12T03:13:42.935482+0000","last_unstale":"2024-03-12T03:14:16.509390+0000","last_undegraded":"2024-03-12T03:14:16.509390+0000","last_fullsized":"2024-03-12T03:14:16.509390+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:43:08.609559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510989+0000","last_change":"2024-03-12T03:13:38.926229+0000","last_active":"2024-03-12T03:14:16.510989+0000","last_peered":"2024-03-12T03:14:16.510989+0000","last_clean":"2024-03-12T03:14:16.510989+0000","last_became_active":"2024-03-12T03:13:38.925917+0000","last_became_peered":"2024-03-12T03:13:38.925917+0000","last_unstale":"2024-03-12T03:14:16.510989+0000","last_undegraded":"2024-03-12T03:14:16.510989+0000","last_fullsized":"2024-03-12T03:14:16.510989+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:07:13.015269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510798+0000","last_change":"2024-03-12T03:13:40.924138+0000","last_active":"2024-03-12T03:14:16.510798+0000","last_peered":"2024-03-12T03:14:16.510798+0000","last_clean":"2024-03-12T03:14:16.510798+0000","last_became_active":"2024-03-12T03:13:40.923266+0000","last_became_peered":"2024-03-12T03:13:40.923266+0000","last_unstale":"2024-03-12T03:14:16.510798+0000","last_undegraded":"2024-03-12T03:14:16.510798+0000","last_fullsized":"2024-03-12T03:14:16.510798+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:08:29.879033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999428+0000","last_change":"2024-03-12T03:13:44.950883+0000","last_active":"2024-03-12T03:14:16.999428+0000","last_peered":"2024-03-12T03:14:16.999428+0000","last_clean":"2024-03-12T03:14:16.999428+0000","last_became_active":"2024-03-12T03:13:44.950518+0000","last_became_peered":"2024-03-12T03:13:44.950518+0000","last_unstale":"2024-03-12T03:14:16.999428+0000","last_undegraded":"2024-03-12T03:14:16.999428+0000","last_fullsized":"2024-03-12T03:14:16.999428+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:03:18.832128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.982035+0000","last_change":"2024-03-12T03:13:42.937176+0000","last_active":"2024-03-12T03:14:53.982035+0000","last_peered":"2024-03-12T03:14:53.982035+0000","last_clean":"2024-03-12T03:14:53.982035+0000","last_became_active":"2024-03-12T03:13:42.936829+0000","last_became_peered":"2024-03-12T03:13:42.936829+0000","last_unstale":"2024-03-12T03:14:53.982035+0000","last_undegraded":"2024-03-12T03:14:53.982035+0000","last_fullsized":"2024-03-12T03:14:53.982035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:00:14.343096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627210+0000","last_change":"2024-03-12T03:13:38.929956+0000","last_active":"2024-03-12T03:14:16.627210+0000","last_peered":"2024-03-12T03:14:16.627210+0000","last_clean":"2024-03-12T03:14:16.627210+0000","last_became_active":"2024-03-12T03:13:38.929652+0000","last_became_peered":"2024-03-12T03:13:38.929652+0000","last_unstale":"2024-03-12T03:14:16.627210+0000","last_undegraded":"2024-03-12T03:14:16.627210+0000","last_fullsized":"2024-03-12T03:14:16.627210+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:55:18.867286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'22","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510770+0000","last_change":"2024-03-12T03:13:41.097345+0000","last_active":"2024-03-12T03:14:16.510770+0000","last_peered":"2024-03-12T03:14:16.510770+0000","last_clean":"2024-03-12T03:14:16.510770+0000","last_became_active":"2024-03-12T03:13:41.097015+0000","last_became_peered":"2024-03-12T03:13:41.097015+0000","last_unstale":"2024-03-12T03:14:16.510770+0000","last_undegraded":"2024-03-12T03:14:16.510770+0000","last_fullsized":"2024-03-12T03:14:16.510770+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:15:35.016446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.511292+0000","last_change":"2024-03-12T03:13:44.947715+0000","last_active":"2024-03-12T03:14:16.511292+0000","last_peered":"2024-03-12T03:14:16.511292+0000","last_clean":"2024-03-12T03:14:16.511292+0000","last_became_active":"2024-03-12T03:13:44.947374+0000","last_became_peered":"2024-03-12T03:13:44.947374+0000","last_unstale":"2024-03-12T03:14:16.511292+0000","last_undegraded":"2024-03-12T03:14:16.511292+0000","last_fullsized":"2024-03-12T03:14:16.511292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:57:22.608130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.982013+0000","last_change":"2024-03-12T03:13:42.937374+0000","last_active":"2024-03-12T03:14:53.982013+0000","last_peered":"2024-03-12T03:14:53.982013+0000","last_clean":"2024-03-12T03:14:53.982013+0000","last_became_active":"2024-03-12T03:13:42.936444+0000","last_became_peered":"2024-03-12T03:13:42.936444+0000","last_unstale":"2024-03-12T03:14:53.982013+0000","last_undegraded":"2024-03-12T03:14:53.982013+0000","last_fullsized":"2024-03-12T03:14:53.982013+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:47:01.032256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999091+0000","last_change":"2024-03-12T03:13:38.924248+0000","last_active":"2024-03-12T03:14:16.999091+0000","last_peered":"2024-03-12T03:14:16.999091+0000","last_clean":"2024-03-12T03:14:16.999091+0000","last_became_active":"2024-03-12T03:13:38.923957+0000","last_became_peered":"2024-03-12T03:13:38.923957+0000","last_unstale":"2024-03-12T03:14:16.999091+0000","last_undegraded":"2024-03-12T03:14:16.999091+0000","last_fullsized":"2024-03-12T03:14:16.999091+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:26:00.742528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'15","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615965+0000","last_change":"2024-03-12T03:13:41.097730+0000","last_active":"2024-03-12T03:14:16.615965+0000","last_peered":"2024-03-12T03:14:16.615965+0000","last_clean":"2024-03-12T03:14:16.615965+0000","last_became_active":"2024-03-12T03:13:41.097368+0000","last_became_peered":"2024-03-12T03:13:41.097368+0000","last_unstale":"2024-03-12T03:14:16.615965+0000","last_undegraded":"2024-03-12T03:14:16.615965+0000","last_fullsized":"2024-03-12T03:14:16.615965+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:29:19.105913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.625206+0000","last_change":"2024-03-12T03:13:44.946631+0000","last_active":"2024-03-12T03:14:16.625206+0000","last_peered":"2024-03-12T03:14:16.625206+0000","last_clean":"2024-03-12T03:14:16.625206+0000","last_became_active":"2024-03-12T03:13:44.946314+0000","last_became_peered":"2024-03-12T03:13:44.946314+0000","last_unstale":"2024-03-12T03:14:16.625206+0000","last_undegraded":"2024-03-12T03:14:16.625206+0000","last_fullsized":"2024-03-12T03:14:16.625206+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:17:12.123391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.512342+0000","last_change":"2024-03-12T03:13:42.938275+0000","last_active":"2024-03-12T03:14:16.512342+0000","last_peered":"2024-03-12T03:14:16.512342+0000","last_clean":"2024-03-12T03:14:16.512342+0000","last_became_active":"2024-03-12T03:13:42.937817+0000","last_became_peered":"2024-03-12T03:13:42.937817+0000","last_unstale":"2024-03-12T03:14:16.512342+0000","last_undegraded":"2024-03-12T03:14:16.512342+0000","last_fullsized":"2024-03-12T03:14:16.512342+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:43:41.883229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627057+0000","last_change":"2024-03-12T03:13:38.921312+0000","last_active":"2024-03-12T03:14:16.627057+0000","last_peered":"2024-03-12T03:14:16.627057+0000","last_clean":"2024-03-12T03:14:16.627057+0000","last_became_active":"2024-03-12T03:13:38.921028+0000","last_became_peered":"2024-03-12T03:13:38.921028+0000","last_unstale":"2024-03-12T03:14:16.627057+0000","last_undegraded":"2024-03-12T03:14:16.627057+0000","last_fullsized":"2024-03-12T03:14:16.627057+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:47:47.532087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'23","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000230+0000","last_change":"2024-03-12T03:13:41.098535+0000","last_active":"2024-03-12T03:14:17.000230+0000","last_peered":"2024-03-12T03:14:17.000230+0000","last_clean":"2024-03-12T03:14:17.000230+0000","last_became_active":"2024-03-12T03:13:41.098250+0000","last_became_peered":"2024-03-12T03:13:41.098250+0000","last_unstale":"2024-03-12T03:14:17.000230+0000","last_undegraded":"2024-03-12T03:14:17.000230+0000","last_fullsized":"2024-03-12T03:14:17.000230+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:10:03.036836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627273+0000","last_change":"2024-03-12T03:13:44.997523+0000","last_active":"2024-03-12T03:14:16.627273+0000","last_peered":"2024-03-12T03:14:16.627273+0000","last_clean":"2024-03-12T03:14:16.627273+0000","last_became_active":"2024-03-12T03:13:44.997227+0000","last_became_peered":"2024-03-12T03:13:44.997227+0000","last_unstale":"2024-03-12T03:14:16.627273+0000","last_undegraded":"2024-03-12T03:14:16.627273+0000","last_fullsized":"2024-03-12T03:14:16.627273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:05:08.063765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615919+0000","last_change":"2024-03-12T03:13:42.932987+0000","last_active":"2024-03-12T03:14:16.615919+0000","last_peered":"2024-03-12T03:14:16.615919+0000","last_clean":"2024-03-12T03:14:16.615919+0000","last_became_active":"2024-03-12T03:13:42.932660+0000","last_became_peered":"2024-03-12T03:13:42.932660+0000","last_unstale":"2024-03-12T03:14:16.615919+0000","last_undegraded":"2024-03-12T03:14:16.615919+0000","last_fullsized":"2024-03-12T03:14:16.615919+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:20.436208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.998805+0000","last_change":"2024-03-12T03:13:38.918748+0000","last_active":"2024-03-12T03:14:16.998805+0000","last_peered":"2024-03-12T03:14:16.998805+0000","last_clean":"2024-03-12T03:14:16.998805+0000","last_became_active":"2024-03-12T03:13:38.918453+0000","last_became_peered":"2024-03-12T03:13:38.918453+0000","last_unstale":"2024-03-12T03:14:16.998805+0000","last_undegraded":"2024-03-12T03:14:16.998805+0000","last_fullsized":"2024-03-12T03:14:16.998805+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:36:56.790011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'12","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.512854+0000","last_change":"2024-03-12T03:13:41.097854+0000","last_active":"2024-03-12T03:14:16.512854+0000","last_peered":"2024-03-12T03:14:16.512854+0000","last_clean":"2024-03-12T03:14:16.512854+0000","last_became_active":"2024-03-12T03:13:41.097410+0000","last_became_peered":"2024-03-12T03:13:41.097410+0000","last_unstale":"2024-03-12T03:14:16.512854+0000","last_undegraded":"2024-03-12T03:14:16.512854+0000","last_fullsized":"2024-03-12T03:14:16.512854+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:13.635492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615622+0000","last_change":"2024-03-12T03:13:44.997634+0000","last_active":"2024-03-12T03:14:16.615622+0000","last_peered":"2024-03-12T03:14:16.615622+0000","last_clean":"2024-03-12T03:14:16.615622+0000","last_became_active":"2024-03-12T03:13:44.997278+0000","last_became_peered":"2024-03-12T03:13:44.997278+0000","last_unstale":"2024-03-12T03:14:16.615622+0000","last_undegraded":"2024-03-12T03:14:16.615622+0000","last_fullsized":"2024-03-12T03:14:16.615622+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:12:19.740465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627596+0000","last_change":"2024-03-12T03:13:38.929873+0000","last_active":"2024-03-12T03:14:16.627596+0000","last_peered":"2024-03-12T03:14:16.627596+0000","last_clean":"2024-03-12T03:14:16.627596+0000","last_became_active":"2024-03-12T03:13:38.929583+0000","last_became_peered":"2024-03-12T03:13:38.929583+0000","last_unstale":"2024-03-12T03:14:16.627596+0000","last_undegraded":"2024-03-12T03:14:16.627596+0000","last_fullsized":"2024-03-12T03:14:16.627596+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:49:15.740169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'27","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615373+0000","last_change":"2024-03-12T03:13:41.097793+0000","last_active":"2024-03-12T03:14:16.615373+0000","last_peered":"2024-03-12T03:14:16.615373+0000","last_clean":"2024-03-12T03:14:16.615373+0000","last_became_active":"2024-03-12T03:13:41.097430+0000","last_became_peered":"2024-03-12T03:13:41.097430+0000","last_unstale":"2024-03-12T03:14:16.615373+0000","last_undegraded":"2024-03-12T03:14:16.615373+0000","last_fullsized":"2024-03-12T03:14:16.615373+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:36.711439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.982110+0000","last_change":"2024-03-12T03:13:42.943578+0000","last_active":"2024-03-12T03:14:53.982110+0000","last_peered":"2024-03-12T03:14:53.982110+0000","last_clean":"2024-03-12T03:14:53.982110+0000","last_became_active":"2024-03-12T03:13:42.943319+0000","last_became_peered":"2024-03-12T03:13:42.943319+0000","last_unstale":"2024-03-12T03:14:53.982110+0000","last_undegraded":"2024-03-12T03:14:53.982110+0000","last_fullsized":"2024-03-12T03:14:53.982110+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:24:24.897970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510234+0000","last_change":"2024-03-12T03:13:44.943749+0000","last_active":"2024-03-12T03:14:16.510234+0000","last_peered":"2024-03-12T03:14:16.510234+0000","last_clean":"2024-03-12T03:14:16.510234+0000","last_became_active":"2024-03-12T03:13:44.943350+0000","last_became_peered":"2024-03-12T03:13:44.943350+0000","last_unstale":"2024-03-12T03:14:16.510234+0000","last_undegraded":"2024-03-12T03:14:16.510234+0000","last_fullsized":"2024-03-12T03:14:16.510234+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:04:52.403194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627494+0000","last_change":"2024-03-12T03:13:38.930076+0000","last_active":"2024-03-12T03:14:16.627494+0000","last_peered":"2024-03-12T03:14:16.627494+0000","last_clean":"2024-03-12T03:14:16.627494+0000","last_became_active":"2024-03-12T03:13:38.929785+0000","last_became_peered":"2024-03-12T03:13:38.929785+0000","last_unstale":"2024-03-12T03:14:16.627494+0000","last_undegraded":"2024-03-12T03:14:16.627494+0000","last_fullsized":"2024-03-12T03:14:16.627494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:31:45.008883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'24","reported_seq":98,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510370+0000","last_change":"2024-03-12T03:14:04.018882+0000","last_active":"2024-03-12T03:14:16.510370+0000","last_peered":"2024-03-12T03:14:16.510370+0000","last_clean":"2024-03-12T03:14:16.510370+0000","last_became_active":"2024-03-12T03:14:04.005976+0000","last_became_peered":"2024-03-12T03:14:04.005976+0000","last_unstale":"2024-03-12T03:14:16.510370+0000","last_undegraded":"2024-03-12T03:14:16.510370+0000","last_fullsized":"2024-03-12T03:14:16.510370+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:39:52.513477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.623642+0000","last_change":"2024-03-12T03:13:42.930881+0000","last_active":"2024-03-12T03:14:16.623642+0000","last_peered":"2024-03-12T03:14:16.623642+0000","last_clean":"2024-03-12T03:14:16.623642+0000","last_became_active":"2024-03-12T03:13:42.930558+0000","last_became_peered":"2024-03-12T03:13:42.930558+0000","last_unstale":"2024-03-12T03:14:16.623642+0000","last_undegraded":"2024-03-12T03:14:16.623642+0000","last_fullsized":"2024-03-12T03:14:16.623642+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:01.690017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.626850+0000","last_change":"2024-03-12T03:13:44.946239+0000","last_active":"2024-03-12T03:14:16.626850+0000","last_peered":"2024-03-12T03:14:16.626850+0000","last_clean":"2024-03-12T03:14:16.626850+0000","last_became_active":"2024-03-12T03:13:44.945891+0000","last_became_peered":"2024-03-12T03:13:44.945891+0000","last_unstale":"2024-03-12T03:14:16.626850+0000","last_undegraded":"2024-03-12T03:14:16.626850+0000","last_fullsized":"2024-03-12T03:14:16.626850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:06:53.343828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.624145+0000","last_change":"2024-03-12T03:13:42.933257+0000","last_active":"2024-03-12T03:14:16.624145+0000","last_peered":"2024-03-12T03:14:16.624145+0000","last_clean":"2024-03-12T03:14:16.624145+0000","last_became_active":"2024-03-12T03:13:42.932919+0000","last_became_peered":"2024-03-12T03:13:42.932919+0000","last_unstale":"2024-03-12T03:14:16.624145+0000","last_undegraded":"2024-03-12T03:14:16.624145+0000","last_fullsized":"2024-03-12T03:14:16.624145+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:22.916168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627680+0000","last_change":"2024-03-12T03:13:38.927221+0000","last_active":"2024-03-12T03:14:16.627680+0000","last_peered":"2024-03-12T03:14:16.627680+0000","last_clean":"2024-03-12T03:14:16.627680+0000","last_became_active":"2024-03-12T03:13:38.926918+0000","last_became_peered":"2024-03-12T03:13:38.926918+0000","last_unstale":"2024-03-12T03:14:16.627680+0000","last_undegraded":"2024-03-12T03:14:16.627680+0000","last_fullsized":"2024-03-12T03:14:16.627680+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:44:56.107368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'25","reported_seq":100,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999413+0000","last_change":"2024-03-12T03:14:04.191395+0000","last_active":"2024-03-12T03:14:16.999413+0000","last_peered":"2024-03-12T03:14:16.999413+0000","last_clean":"2024-03-12T03:14:16.999413+0000","last_became_active":"2024-03-12T03:14:04.180406+0000","last_became_peered":"2024-03-12T03:14:04.180406+0000","last_unstale":"2024-03-12T03:14:16.999413+0000","last_undegraded":"2024-03-12T03:14:16.999413+0000","last_fullsized":"2024-03-12T03:14:16.999413+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:01:27.932066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999018+0000","last_change":"2024-03-12T03:13:44.948367+0000","last_active":"2024-03-12T03:14:16.999018+0000","last_peered":"2024-03-12T03:14:16.999018+0000","last_clean":"2024-03-12T03:14:16.999018+0000","last_became_active":"2024-03-12T03:13:44.948026+0000","last_became_peered":"2024-03-12T03:13:44.948026+0000","last_unstale":"2024-03-12T03:14:16.999018+0000","last_undegraded":"2024-03-12T03:14:16.999018+0000","last_fullsized":"2024-03-12T03:14:16.999018+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:50:55.995044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.623809+0000","last_change":"2024-03-12T03:13:42.930758+0000","last_active":"2024-03-12T03:14:16.623809+0000","last_peered":"2024-03-12T03:14:16.623809+0000","last_clean":"2024-03-12T03:14:16.623809+0000","last_became_active":"2024-03-12T03:13:42.930418+0000","last_became_peered":"2024-03-12T03:13:42.930418+0000","last_unstale":"2024-03-12T03:14:16.623809+0000","last_undegraded":"2024-03-12T03:14:16.623809+0000","last_fullsized":"2024-03-12T03:14:16.623809+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:54:48.011915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615673+0000","last_change":"2024-03-12T03:13:38.923640+0000","last_active":"2024-03-12T03:14:16.615673+0000","last_peered":"2024-03-12T03:14:16.615673+0000","last_clean":"2024-03-12T03:14:16.615673+0000","last_became_active":"2024-03-12T03:13:38.923299+0000","last_became_peered":"2024-03-12T03:13:38.923299+0000","last_unstale":"2024-03-12T03:14:16.615673+0000","last_undegraded":"2024-03-12T03:14:16.615673+0000","last_fullsized":"2024-03-12T03:14:16.615673+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:33:53.613151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'15","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.001049+0000","last_change":"2024-03-12T03:13:40.933773+0000","last_active":"2024-03-12T03:14:17.001049+0000","last_peered":"2024-03-12T03:14:17.001049+0000","last_clean":"2024-03-12T03:14:17.001049+0000","last_became_active":"2024-03-12T03:13:40.933262+0000","last_became_peered":"2024-03-12T03:13:40.933262+0000","last_unstale":"2024-03-12T03:14:17.001049+0000","last_undegraded":"2024-03-12T03:14:17.001049+0000","last_fullsized":"2024-03-12T03:14:17.001049+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:28:49.669840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.511458+0000","last_change":"2024-03-12T03:13:44.944810+0000","last_active":"2024-03-12T03:14:16.511458+0000","last_peered":"2024-03-12T03:14:16.511458+0000","last_clean":"2024-03-12T03:14:16.511458+0000","last_became_active":"2024-03-12T03:13:44.943384+0000","last_became_peered":"2024-03-12T03:13:44.943384+0000","last_unstale":"2024-03-12T03:14:16.511458+0000","last_undegraded":"2024-03-12T03:14:16.511458+0000","last_fullsized":"2024-03-12T03:14:16.511458+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:17.957042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615461+0000","last_change":"2024-03-12T03:13:41.921832+0000","last_active":"2024-03-12T03:14:16.615461+0000","last_peered":"2024-03-12T03:14:16.615461+0000","last_clean":"2024-03-12T03:14:16.615461+0000","last_became_active":"2024-03-12T03:13:37.916764+0000","last_became_peered":"2024-03-12T03:13:37.916764+0000","last_unstale":"2024-03-12T03:14:16.615461+0000","last_undegraded":"2024-03-12T03:14:16.615461+0000","last_fullsized":"2024-03-12T03:14:16.615461+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:36.900834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:36.900834+0000","last_clean_scrub_stamp":"2024-03-12T03:13:36.900834+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:01.955380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035682550000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510416+0000","last_change":"2024-03-12T03:13:38.921875+0000","last_active":"2024-03-12T03:14:16.510416+0000","last_peered":"2024-03-12T03:14:16.510416+0000","last_clean":"2024-03-12T03:14:16.510416+0000","last_became_active":"2024-03-12T03:13:38.921588+0000","last_became_peered":"2024-03-12T03:13:38.921588+0000","last_unstale":"2024-03-12T03:14:16.510416+0000","last_undegraded":"2024-03-12T03:14:16.510416+0000","last_fullsized":"2024-03-12T03:14:16.510416+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:07:57.296831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'34","reported_seq":106,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.981737+0000","last_change":"2024-03-12T03:13:40.929007+0000","last_active":"2024-03-12T03:14:53.981737+0000","last_peered":"2024-03-12T03:14:53.981737+0000","last_clean":"2024-03-12T03:14:53.981737+0000","last_became_active":"2024-03-12T03:13:40.928584+0000","last_became_peered":"2024-03-12T03:13:40.928584+0000","last_unstale":"2024-03-12T03:14:53.981737+0000","last_undegraded":"2024-03-12T03:14:53.981737+0000","last_fullsized":"2024-03-12T03:14:53.981737+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:46:43.916913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999906+0000","last_change":"2024-03-12T03:13:42.930916+0000","last_active":"2024-03-12T03:14:16.999906+0000","last_peered":"2024-03-12T03:14:16.999906+0000","last_clean":"2024-03-12T03:14:16.999906+0000","last_became_active":"2024-03-12T03:13:42.930028+0000","last_became_peered":"2024-03-12T03:13:42.930028+0000","last_unstale":"2024-03-12T03:14:16.999906+0000","last_undegraded":"2024-03-12T03:14:16.999906+0000","last_fullsized":"2024-03-12T03:14:16.999906+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:59:16.700540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627860+0000","last_change":"2024-03-12T03:13:44.998674+0000","last_active":"2024-03-12T03:14:16.627860+0000","last_peered":"2024-03-12T03:14:16.627860+0000","last_clean":"2024-03-12T03:14:16.627860+0000","last_became_active":"2024-03-12T03:13:44.998360+0000","last_became_peered":"2024-03-12T03:13:44.998360+0000","last_unstale":"2024-03-12T03:14:16.627860+0000","last_undegraded":"2024-03-12T03:14:16.627860+0000","last_fullsized":"2024-03-12T03:14:16.627860+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:55:34.393299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.508782+0000","last_change":"2024-03-12T03:13:38.919604+0000","last_active":"2024-03-12T03:14:16.508782+0000","last_peered":"2024-03-12T03:14:16.508782+0000","last_clean":"2024-03-12T03:14:16.508782+0000","last_became_active":"2024-03-12T03:13:38.918065+0000","last_became_peered":"2024-03-12T03:13:38.918065+0000","last_unstale":"2024-03-12T03:14:16.508782+0000","last_undegraded":"2024-03-12T03:14:16.508782+0000","last_fullsized":"2024-03-12T03:14:16.508782+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:33:12.232739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.624200+0000","last_change":"2024-03-12T03:13:41.916325+0000","last_active":"2024-03-12T03:14:16.624200+0000","last_peered":"2024-03-12T03:14:16.624200+0000","last_clean":"2024-03-12T03:14:16.624200+0000","last_became_active":"2024-03-12T03:13:37.917610+0000","last_became_peered":"2024-03-12T03:13:37.917610+0000","last_unstale":"2024-03-12T03:14:16.624200+0000","last_undegraded":"2024-03-12T03:14:16.624200+0000","last_fullsized":"2024-03-12T03:14:16.624200+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:36.900834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:36.900834+0000","last_clean_scrub_stamp":"2024-03-12T03:13:36.900834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:53:54.666848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055126199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'19","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.511530+0000","last_change":"2024-03-12T03:13:40.927880+0000","last_active":"2024-03-12T03:14:16.511530+0000","last_peered":"2024-03-12T03:14:16.511530+0000","last_clean":"2024-03-12T03:14:16.511530+0000","last_became_active":"2024-03-12T03:13:40.924923+0000","last_became_peered":"2024-03-12T03:13:40.924923+0000","last_unstale":"2024-03-12T03:14:16.511530+0000","last_undegraded":"2024-03-12T03:14:16.511530+0000","last_fullsized":"2024-03-12T03:14:16.511530+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:51:51.830745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.624338+0000","last_change":"2024-03-12T03:13:42.935113+0000","last_active":"2024-03-12T03:14:16.624338+0000","last_peered":"2024-03-12T03:14:16.624338+0000","last_clean":"2024-03-12T03:14:16.624338+0000","last_became_active":"2024-03-12T03:13:42.934797+0000","last_became_peered":"2024-03-12T03:13:42.934797+0000","last_unstale":"2024-03-12T03:14:16.624338+0000","last_undegraded":"2024-03-12T03:14:16.624338+0000","last_fullsized":"2024-03-12T03:14:16.624338+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:49:17.115408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615886+0000","last_change":"2024-03-12T03:13:44.940697+0000","last_active":"2024-03-12T03:14:16.615886+0000","last_peered":"2024-03-12T03:14:16.615886+0000","last_clean":"2024-03-12T03:14:16.615886+0000","last_became_active":"2024-03-12T03:13:44.940391+0000","last_became_peered":"2024-03-12T03:13:44.940391+0000","last_unstale":"2024-03-12T03:14:16.615886+0000","last_undegraded":"2024-03-12T03:14:16.615886+0000","last_fullsized":"2024-03-12T03:14:16.615886+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:48:46.696099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"72'128","reported_seq":278,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:18.628450+0000","last_change":"2024-03-12T03:13:28.492572+0000","last_active":"2024-03-12T03:14:18.628450+0000","last_peered":"2024-03-12T03:14:18.628450+0000","last_clean":"2024-03-12T03:14:18.628450+0000","last_became_active":"2024-03-12T03:13:28.492056+0000","last_became_peered":"2024-03-12T03:13:28.492056+0000","last_unstale":"2024-03-12T03:14:18.628450+0000","last_undegraded":"2024-03-12T03:14:18.628450+0000","last_fullsized":"2024-03-12T03:14:18.628450+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:11:21.093480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:11:21.093480+0000","last_clean_scrub_stamp":"2024-03-12T03:11:21.093480+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:59:57.354004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509972+0000","last_change":"2024-03-12T03:13:38.927650+0000","last_active":"2024-03-12T03:14:16.509972+0000","last_peered":"2024-03-12T03:14:16.509972+0000","last_clean":"2024-03-12T03:14:16.509972+0000","last_became_active":"2024-03-12T03:13:38.927310+0000","last_became_peered":"2024-03-12T03:13:38.927310+0000","last_unstale":"2024-03-12T03:14:16.509972+0000","last_undegraded":"2024-03-12T03:14:16.509972+0000","last_fullsized":"2024-03-12T03:14:16.509972+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:38:20.477500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.982205+0000","last_change":"2024-03-12T03:13:41.098065+0000","last_active":"2024-03-12T03:14:53.982205+0000","last_peered":"2024-03-12T03:14:53.982205+0000","last_clean":"2024-03-12T03:14:53.982205+0000","last_became_active":"2024-03-12T03:13:41.097186+0000","last_became_peered":"2024-03-12T03:13:41.097186+0000","last_unstale":"2024-03-12T03:14:53.982205+0000","last_undegraded":"2024-03-12T03:14:53.982205+0000","last_fullsized":"2024-03-12T03:14:53.982205+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:42:22.670132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627755+0000","last_change":"2024-03-12T03:13:42.940542+0000","last_active":"2024-03-12T03:14:16.627755+0000","last_peered":"2024-03-12T03:14:16.627755+0000","last_clean":"2024-03-12T03:14:16.627755+0000","last_became_active":"2024-03-12T03:13:42.940245+0000","last_became_peered":"2024-03-12T03:13:42.940245+0000","last_unstale":"2024-03-12T03:14:16.627755+0000","last_undegraded":"2024-03-12T03:14:16.627755+0000","last_fullsized":"2024-03-12T03:14:16.627755+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:01.095734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510001+0000","last_change":"2024-03-12T03:13:44.949096+0000","last_active":"2024-03-12T03:14:16.510001+0000","last_peered":"2024-03-12T03:14:16.510001+0000","last_clean":"2024-03-12T03:14:16.510001+0000","last_became_active":"2024-03-12T03:13:44.948108+0000","last_became_peered":"2024-03-12T03:13:44.948108+0000","last_unstale":"2024-03-12T03:14:16.510001+0000","last_undegraded":"2024-03-12T03:14:16.510001+0000","last_fullsized":"2024-03-12T03:14:16.510001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:04:58.946629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999310+0000","last_change":"2024-03-12T03:13:38.924200+0000","last_active":"2024-03-12T03:14:16.999310+0000","last_peered":"2024-03-12T03:14:16.999310+0000","last_clean":"2024-03-12T03:14:16.999310+0000","last_became_active":"2024-03-12T03:13:38.923882+0000","last_became_peered":"2024-03-12T03:13:38.923882+0000","last_unstale":"2024-03-12T03:14:16.999310+0000","last_undegraded":"2024-03-12T03:14:16.999310+0000","last_fullsized":"2024-03-12T03:14:16.999310+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:29:21.228313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":122,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:15:00.812449+0000","last_change":"2024-03-12T03:13:41.922084+0000","last_active":"2024-03-12T03:15:00.812449+0000","last_peered":"2024-03-12T03:15:00.812449+0000","last_clean":"2024-03-12T03:15:00.812449+0000","last_became_active":"2024-03-12T03:13:37.917516+0000","last_became_peered":"2024-03-12T03:13:37.917516+0000","last_unstale":"2024-03-12T03:15:00.812449+0000","last_undegraded":"2024-03-12T03:15:00.812449+0000","last_fullsized":"2024-03-12T03:15:00.812449+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:36.900834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:36.900834+0000","last_clean_scrub_stamp":"2024-03-12T03:13:36.900834+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:30:40.218046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000451154,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":77,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":47,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000071+0000","last_change":"2024-03-12T03:13:40.932547+0000","last_active":"2024-03-12T03:14:17.000071+0000","last_peered":"2024-03-12T03:14:17.000071+0000","last_clean":"2024-03-12T03:14:17.000071+0000","last_became_active":"2024-03-12T03:13:40.932123+0000","last_became_peered":"2024-03-12T03:13:40.932123+0000","last_unstale":"2024-03-12T03:14:17.000071+0000","last_undegraded":"2024-03-12T03:14:17.000071+0000","last_fullsized":"2024-03-12T03:14:17.000071+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:28:59.215655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.616097+0000","last_change":"2024-03-12T03:13:42.937296+0000","last_active":"2024-03-12T03:14:16.616097+0000","last_peered":"2024-03-12T03:14:16.616097+0000","last_clean":"2024-03-12T03:14:16.616097+0000","last_became_active":"2024-03-12T03:13:42.937033+0000","last_became_peered":"2024-03-12T03:13:42.937033+0000","last_unstale":"2024-03-12T03:14:16.616097+0000","last_undegraded":"2024-03-12T03:14:16.616097+0000","last_fullsized":"2024-03-12T03:14:16.616097+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:39:04.707241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.508812+0000","last_change":"2024-03-12T03:13:44.997442+0000","last_active":"2024-03-12T03:14:16.508812+0000","last_peered":"2024-03-12T03:14:16.508812+0000","last_clean":"2024-03-12T03:14:16.508812+0000","last_became_active":"2024-03-12T03:13:44.997110+0000","last_became_peered":"2024-03-12T03:13:44.997110+0000","last_unstale":"2024-03-12T03:14:16.508812+0000","last_undegraded":"2024-03-12T03:14:16.508812+0000","last_fullsized":"2024-03-12T03:14:16.508812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:30:38.116899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510218+0000","last_change":"2024-03-12T03:13:38.917009+0000","last_active":"2024-03-12T03:14:16.510218+0000","last_peered":"2024-03-12T03:14:16.510218+0000","last_clean":"2024-03-12T03:14:16.510218+0000","last_became_active":"2024-03-12T03:13:38.916752+0000","last_became_peered":"2024-03-12T03:13:38.916752+0000","last_unstale":"2024-03-12T03:14:16.510218+0000","last_undegraded":"2024-03-12T03:14:16.510218+0000","last_fullsized":"2024-03-12T03:14:16.510218+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:59:37.556237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'25","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000862+0000","last_change":"2024-03-12T03:13:41.097510+0000","last_active":"2024-03-12T03:14:17.000862+0000","last_peered":"2024-03-12T03:14:17.000862+0000","last_clean":"2024-03-12T03:14:17.000862+0000","last_became_active":"2024-03-12T03:13:41.097205+0000","last_became_peered":"2024-03-12T03:13:41.097205+0000","last_unstale":"2024-03-12T03:14:17.000862+0000","last_undegraded":"2024-03-12T03:14:17.000862+0000","last_fullsized":"2024-03-12T03:14:17.000862+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:43:52.874518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999420+0000","last_change":"2024-03-12T03:13:42.936931+0000","last_active":"2024-03-12T03:14:16.999420+0000","last_peered":"2024-03-12T03:14:16.999420+0000","last_clean":"2024-03-12T03:14:16.999420+0000","last_became_active":"2024-03-12T03:13:42.936643+0000","last_became_peered":"2024-03-12T03:13:42.936643+0000","last_unstale":"2024-03-12T03:14:16.999420+0000","last_undegraded":"2024-03-12T03:14:16.999420+0000","last_fullsized":"2024-03-12T03:14:16.999420+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:57.012793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999577+0000","last_change":"2024-03-12T03:13:44.950852+0000","last_active":"2024-03-12T03:14:16.999577+0000","last_peered":"2024-03-12T03:14:16.999577+0000","last_clean":"2024-03-12T03:14:16.999577+0000","last_became_active":"2024-03-12T03:13:44.950501+0000","last_became_peered":"2024-03-12T03:13:44.950501+0000","last_unstale":"2024-03-12T03:14:16.999577+0000","last_undegraded":"2024-03-12T03:14:16.999577+0000","last_fullsized":"2024-03-12T03:14:16.999577+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:31:14.165775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615843+0000","last_change":"2024-03-12T03:13:38.925086+0000","last_active":"2024-03-12T03:14:16.615843+0000","last_peered":"2024-03-12T03:14:16.615843+0000","last_clean":"2024-03-12T03:14:16.615843+0000","last_became_active":"2024-03-12T03:13:38.924772+0000","last_became_peered":"2024-03-12T03:13:38.924772+0000","last_unstale":"2024-03-12T03:14:16.615843+0000","last_undegraded":"2024-03-12T03:14:16.615843+0000","last_fullsized":"2024-03-12T03:14:16.615843+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:51:26.322445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'16","reported_seq":56,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509465+0000","last_change":"2024-03-12T03:13:40.922975+0000","last_active":"2024-03-12T03:14:16.509465+0000","last_peered":"2024-03-12T03:14:16.509465+0000","last_clean":"2024-03-12T03:14:16.509465+0000","last_became_active":"2024-03-12T03:13:40.922569+0000","last_became_peered":"2024-03-12T03:13:40.922569+0000","last_unstale":"2024-03-12T03:14:16.509465+0000","last_undegraded":"2024-03-12T03:14:16.509465+0000","last_fullsized":"2024-03-12T03:14:16.509465+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:29:06.817123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.981736+0000","last_change":"2024-03-12T03:13:42.935058+0000","last_active":"2024-03-12T03:14:53.981736+0000","last_peered":"2024-03-12T03:14:53.981736+0000","last_clean":"2024-03-12T03:14:53.981736+0000","last_became_active":"2024-03-12T03:13:42.934725+0000","last_became_peered":"2024-03-12T03:13:42.934725+0000","last_unstale":"2024-03-12T03:14:53.981736+0000","last_undegraded":"2024-03-12T03:14:53.981736+0000","last_fullsized":"2024-03-12T03:14:53.981736+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:47:43.360515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.627343+0000","last_change":"2024-03-12T03:13:44.997517+0000","last_active":"2024-03-12T03:14:16.627343+0000","last_peered":"2024-03-12T03:14:16.627343+0000","last_clean":"2024-03-12T03:14:16.627343+0000","last_became_active":"2024-03-12T03:13:44.997190+0000","last_became_peered":"2024-03-12T03:13:44.997190+0000","last_unstale":"2024-03-12T03:14:16.627343+0000","last_undegraded":"2024-03-12T03:14:16.627343+0000","last_fullsized":"2024-03-12T03:14:16.627343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:00:05.360461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:17.000102+0000","last_change":"2024-03-12T03:13:38.919557+0000","last_active":"2024-03-12T03:14:17.000102+0000","last_peered":"2024-03-12T03:14:17.000102+0000","last_clean":"2024-03-12T03:14:17.000102+0000","last_became_active":"2024-03-12T03:13:38.919278+0000","last_became_peered":"2024-03-12T03:13:38.919278+0000","last_unstale":"2024-03-12T03:14:17.000102+0000","last_undegraded":"2024-03-12T03:14:17.000102+0000","last_fullsized":"2024-03-12T03:14:17.000102+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:03:51.500359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'20","reported_seq":62,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510177+0000","last_change":"2024-03-12T03:13:40.923874+0000","last_active":"2024-03-12T03:14:16.510177+0000","last_peered":"2024-03-12T03:14:16.510177+0000","last_clean":"2024-03-12T03:14:16.510177+0000","last_became_active":"2024-03-12T03:13:40.923540+0000","last_became_peered":"2024-03-12T03:13:40.923540+0000","last_unstale":"2024-03-12T03:14:16.510177+0000","last_undegraded":"2024-03-12T03:14:16.510177+0000","last_fullsized":"2024-03-12T03:14:16.510177+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:39:53.308593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.981735+0000","last_change":"2024-03-12T03:13:42.933668+0000","last_active":"2024-03-12T03:14:53.981735+0000","last_peered":"2024-03-12T03:14:53.981735+0000","last_clean":"2024-03-12T03:14:53.981735+0000","last_became_active":"2024-03-12T03:13:42.932841+0000","last_became_peered":"2024-03-12T03:13:42.932841+0000","last_unstale":"2024-03-12T03:14:53.981735+0000","last_undegraded":"2024-03-12T03:14:53.981735+0000","last_fullsized":"2024-03-12T03:14:53.981735+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:18:50.441610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510248+0000","last_change":"2024-03-12T03:13:44.947738+0000","last_active":"2024-03-12T03:14:16.510248+0000","last_peered":"2024-03-12T03:14:16.510248+0000","last_clean":"2024-03-12T03:14:16.510248+0000","last_became_active":"2024-03-12T03:13:44.947430+0000","last_became_peered":"2024-03-12T03:13:44.947430+0000","last_unstale":"2024-03-12T03:14:16.510248+0000","last_undegraded":"2024-03-12T03:14:16.510248+0000","last_fullsized":"2024-03-12T03:14:16.510248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:30:57.566324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.512111+0000","last_change":"2024-03-12T03:13:38.929141+0000","last_active":"2024-03-12T03:14:16.512111+0000","last_peered":"2024-03-12T03:14:16.512111+0000","last_clean":"2024-03-12T03:14:16.512111+0000","last_became_active":"2024-03-12T03:13:38.928839+0000","last_became_peered":"2024-03-12T03:13:38.928839+0000","last_unstale":"2024-03-12T03:14:16.512111+0000","last_undegraded":"2024-03-12T03:14:16.512111+0000","last_fullsized":"2024-03-12T03:14:16.512111+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:45.937151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'30","reported_seq":85,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509828+0000","last_change":"2024-03-12T03:13:41.097359+0000","last_active":"2024-03-12T03:14:16.509828+0000","last_peered":"2024-03-12T03:14:16.509828+0000","last_clean":"2024-03-12T03:14:16.509828+0000","last_became_active":"2024-03-12T03:13:41.097064+0000","last_became_peered":"2024-03-12T03:13:41.097064+0000","last_unstale":"2024-03-12T03:14:16.509828+0000","last_undegraded":"2024-03-12T03:14:16.509828+0000","last_fullsized":"2024-03-12T03:14:16.509828+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:25:25.853023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509939+0000","last_change":"2024-03-12T03:13:42.934233+0000","last_active":"2024-03-12T03:14:16.509939+0000","last_peered":"2024-03-12T03:14:16.509939+0000","last_clean":"2024-03-12T03:14:16.509939+0000","last_became_active":"2024-03-12T03:13:42.933939+0000","last_became_peered":"2024-03-12T03:13:42.933939+0000","last_unstale":"2024-03-12T03:14:16.509939+0000","last_undegraded":"2024-03-12T03:14:16.509939+0000","last_fullsized":"2024-03-12T03:14:16.509939+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:28.340329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.615691+0000","last_change":"2024-03-12T03:13:44.942867+0000","last_active":"2024-03-12T03:14:16.615691+0000","last_peered":"2024-03-12T03:14:16.615691+0000","last_clean":"2024-03-12T03:14:16.615691+0000","last_became_active":"2024-03-12T03:13:44.942571+0000","last_became_peered":"2024-03-12T03:13:44.942571+0000","last_unstale":"2024-03-12T03:14:16.615691+0000","last_undegraded":"2024-03-12T03:14:16.615691+0000","last_fullsized":"2024-03-12T03:14:16.615691+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:24:41.603955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.982068+0000","last_change":"2024-03-12T03:13:42.932999+0000","last_active":"2024-03-12T03:14:53.982068+0000","last_peered":"2024-03-12T03:14:53.982068+0000","last_clean":"2024-03-12T03:14:53.982068+0000","last_became_active":"2024-03-12T03:13:42.932713+0000","last_became_peered":"2024-03-12T03:13:42.932713+0000","last_unstale":"2024-03-12T03:14:53.982068+0000","last_undegraded":"2024-03-12T03:14:53.982068+0000","last_fullsized":"2024-03-12T03:14:53.982068+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:43:32.261711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509740+0000","last_change":"2024-03-12T03:13:38.925544+0000","last_active":"2024-03-12T03:14:16.509740+0000","last_peered":"2024-03-12T03:14:16.509740+0000","last_clean":"2024-03-12T03:14:16.509740+0000","last_became_active":"2024-03-12T03:13:38.925258+0000","last_became_peered":"2024-03-12T03:13:38.925258+0000","last_unstale":"2024-03-12T03:14:16.509740+0000","last_undegraded":"2024-03-12T03:14:16.509740+0000","last_fullsized":"2024-03-12T03:14:16.509740+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:01:23.224690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.511481+0000","last_change":"2024-03-12T03:13:40.923543+0000","last_active":"2024-03-12T03:14:16.511481+0000","last_peered":"2024-03-12T03:14:16.511481+0000","last_clean":"2024-03-12T03:14:16.511481+0000","last_became_active":"2024-03-12T03:13:40.922942+0000","last_became_peered":"2024-03-12T03:13:40.922942+0000","last_unstale":"2024-03-12T03:14:16.511481+0000","last_undegraded":"2024-03-12T03:14:16.511481+0000","last_fullsized":"2024-03-12T03:14:16.511481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:45:42.789885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510681+0000","last_change":"2024-03-12T03:13:44.997399+0000","last_active":"2024-03-12T03:14:16.510681+0000","last_peered":"2024-03-12T03:14:16.510681+0000","last_clean":"2024-03-12T03:14:16.510681+0000","last_became_active":"2024-03-12T03:13:44.997070+0000","last_became_peered":"2024-03-12T03:13:44.997070+0000","last_unstale":"2024-03-12T03:14:16.510681+0000","last_undegraded":"2024-03-12T03:14:16.510681+0000","last_fullsized":"2024-03-12T03:14:16.510681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:28:03.546726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.616093+0000","last_change":"2024-03-12T03:13:42.939452+0000","last_active":"2024-03-12T03:14:16.616093+0000","last_peered":"2024-03-12T03:14:16.616093+0000","last_clean":"2024-03-12T03:14:16.616093+0000","last_became_active":"2024-03-12T03:13:42.939148+0000","last_became_peered":"2024-03-12T03:13:42.939148+0000","last_unstale":"2024-03-12T03:14:16.616093+0000","last_undegraded":"2024-03-12T03:14:16.616093+0000","last_fullsized":"2024-03-12T03:14:16.616093+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:19:14.944979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.509945+0000","last_change":"2024-03-12T03:13:38.926088+0000","last_active":"2024-03-12T03:14:16.509945+0000","last_peered":"2024-03-12T03:14:16.509945+0000","last_clean":"2024-03-12T03:14:16.509945+0000","last_became_active":"2024-03-12T03:13:38.925616+0000","last_became_peered":"2024-03-12T03:13:38.925616+0000","last_unstale":"2024-03-12T03:14:16.509945+0000","last_undegraded":"2024-03-12T03:14:16.509945+0000","last_fullsized":"2024-03-12T03:14:16.509945+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:37:18.439932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'17","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.511064+0000","last_change":"2024-03-12T03:13:40.924337+0000","last_active":"2024-03-12T03:14:16.511064+0000","last_peered":"2024-03-12T03:14:16.511064+0000","last_clean":"2024-03-12T03:14:16.511064+0000","last_became_active":"2024-03-12T03:13:40.923975+0000","last_became_peered":"2024-03-12T03:13:40.923975+0000","last_unstale":"2024-03-12T03:14:16.511064+0000","last_undegraded":"2024-03-12T03:14:16.511064+0000","last_fullsized":"2024-03-12T03:14:16.511064+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:39:55.761682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.624756+0000","last_change":"2024-03-12T03:13:44.948402+0000","last_active":"2024-03-12T03:14:16.624756+0000","last_peered":"2024-03-12T03:14:16.624756+0000","last_clean":"2024-03-12T03:14:16.624756+0000","last_became_active":"2024-03-12T03:13:44.947990+0000","last_became_peered":"2024-03-12T03:13:44.947990+0000","last_unstale":"2024-03-12T03:14:16.624756+0000","last_undegraded":"2024-03-12T03:14:16.624756+0000","last_fullsized":"2024-03-12T03:14:16.624756+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:07:59.047651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510356+0000","last_change":"2024-03-12T03:13:42.941821+0000","last_active":"2024-03-12T03:14:16.510356+0000","last_peered":"2024-03-12T03:14:16.510356+0000","last_clean":"2024-03-12T03:14:16.510356+0000","last_became_active":"2024-03-12T03:13:42.941472+0000","last_became_peered":"2024-03-12T03:13:42.941472+0000","last_unstale":"2024-03-12T03:14:16.510356+0000","last_undegraded":"2024-03-12T03:14:16.510356+0000","last_fullsized":"2024-03-12T03:14:16.510356+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:02.076746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.999302+0000","last_change":"2024-03-12T03:13:38.923139+0000","last_active":"2024-03-12T03:14:16.999302+0000","last_peered":"2024-03-12T03:14:16.999302+0000","last_clean":"2024-03-12T03:14:16.999302+0000","last_became_active":"2024-03-12T03:13:38.922772+0000","last_became_peered":"2024-03-12T03:13:38.922772+0000","last_unstale":"2024-03-12T03:14:16.999302+0000","last_undegraded":"2024-03-12T03:14:16.999302+0000","last_fullsized":"2024-03-12T03:14:16.999302+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:57.646908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510598+0000","last_change":"2024-03-12T03:13:40.927968+0000","last_active":"2024-03-12T03:14:16.510598+0000","last_peered":"2024-03-12T03:14:16.510598+0000","last_clean":"2024-03-12T03:14:16.510598+0000","last_became_active":"2024-03-12T03:13:40.927705+0000","last_became_peered":"2024-03-12T03:13:40.927705+0000","last_unstale":"2024-03-12T03:14:16.510598+0000","last_undegraded":"2024-03-12T03:14:16.510598+0000","last_fullsized":"2024-03-12T03:14:16.510598+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:53.556755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510974+0000","last_change":"2024-03-12T03:13:44.947632+0000","last_active":"2024-03-12T03:14:16.510974+0000","last_peered":"2024-03-12T03:14:16.510974+0000","last_clean":"2024-03-12T03:14:16.510974+0000","last_became_active":"2024-03-12T03:13:44.947325+0000","last_became_peered":"2024-03-12T03:13:44.947325+0000","last_unstale":"2024-03-12T03:14:16.510974+0000","last_undegraded":"2024-03-12T03:14:16.510974+0000","last_fullsized":"2024-03-12T03:14:16.510974+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:43.919104+0000","last_clean_scrub_stamp":"2024-03-12T03:13:43.919104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:13:26.591561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:53.981741+0000","last_change":"2024-03-12T03:13:42.937388+0000","last_active":"2024-03-12T03:14:53.981741+0000","last_peered":"2024-03-12T03:14:53.981741+0000","last_clean":"2024-03-12T03:14:53.981741+0000","last_became_active":"2024-03-12T03:13:42.937030+0000","last_became_peered":"2024-03-12T03:13:42.937030+0000","last_unstale":"2024-03-12T03:14:53.981741+0000","last_undegraded":"2024-03-12T03:14:53.981741+0000","last_fullsized":"2024-03-12T03:14:53.981741+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:41.912382+0000","last_clean_scrub_stamp":"2024-03-12T03:13:41.912382+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:35:41.695226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510475+0000","last_change":"2024-03-12T03:13:38.926318+0000","last_active":"2024-03-12T03:14:16.510475+0000","last_peered":"2024-03-12T03:14:16.510475+0000","last_clean":"2024-03-12T03:14:16.510475+0000","last_became_active":"2024-03-12T03:13:38.925999+0000","last_became_peered":"2024-03-12T03:13:38.925999+0000","last_unstale":"2024-03-12T03:14:16.510475+0000","last_undegraded":"2024-03-12T03:14:16.510475+0000","last_fullsized":"2024-03-12T03:14:16.510475+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:37.905423+0000","last_clean_scrub_stamp":"2024-03-12T03:13:37.905423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:11:46.388826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-12T03:14:16.510899+0000","last_change":"2024-03-12T03:13:40.924917+0000","last_active":"2024-03-12T03:14:16.510899+0000","last_peered":"2024-03-12T03:14:16.510899+0000","last_clean":"2024-03-12T03:14:16.510899+0000","last_became_active":"2024-03-12T03:13:40.924581+0000","last_became_peered":"2024-03-12T03:13:40.924581+0000","last_unstale":"2024-03-12T03:14:16.510899+0000","last_undegraded":"2024-03-12T03:14:16.510899+0000","last_fullsized":"2024-03-12T03:14:16.510899+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:13:39.908590+0000","last_clean_scrub_stamp":"2024-03-12T03:13:39.908590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:25.698015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":587,"ondisk_log_size":587,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":72,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1314816,"data_stored":872356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712700,"statfs":{"total":95995035648,"available":95961804800,"internally_reserved":0,"allocated":1302528,"data_stored":872651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717408,"statfs":{"total":95995035648,"available":95966625792,"internally_reserved":0,"allocated":741376,"data_stored":282158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32468,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712684,"statfs":{"total":95995035648,"available":95961788416,"internally_reserved":0,"allocated":1323008,"data_stored":873432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":712704,"data_stored":281491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":281814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":724992,"data_stored":281528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":712704,"data_stored":282706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:15:07.688 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-12T03:15:07.689 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-12T03:15:07.689 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-12T03:15:07.689 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 964142e8-e01d-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-12T03:15:07.936 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:15:07.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:07 smithi094 ceph-mon[29291]: from='client.14763 v1:172.21.15.94:0/450834611' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:15:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:07 smithi094 ceph-mon[40484]: from='client.14763 v1:172.21.15.94:0/450834611' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:15:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:07 smithi151 ceph-mon[32923]: from='client.14763 v1:172.21.15.94:0/450834611' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:15:07.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:15:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:15:08.709 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:08 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[75451]: ts=2024-03-12T03:15:08.508Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002713988s 2024-03-12T03:15:08.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:08 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:08.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:08 smithi094 ceph-mon[29291]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:08 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:08 smithi094 ceph-mon[40484]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:08 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:08 smithi151 ceph-mon[32923]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:09.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.607+0000 7fdf95262700 1 -- 172.21.15.94:0/2305905828 >> v1:172.21.15.94:6790/0 conn(0x7fdf9010aa70 legacy=0x7fdf9010ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:09.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.607+0000 7fdf95262700 1 -- 172.21.15.94:0/2305905828 shutdown_connections 2024-03-12T03:15:09.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.607+0000 7fdf95262700 1 -- 172.21.15.94:0/2305905828 >> 172.21.15.94:0/2305905828 conn(0x7fdf900fd690 msgr2=0x7fdf900ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:09.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.607+0000 7fdf95262700 1 -- 172.21.15.94:0/2305905828 shutdown_connections 2024-03-12T03:15:09.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.607+0000 7fdf95262700 1 -- 172.21.15.94:0/2305905828 wait complete. 2024-03-12T03:15:09.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.607+0000 7fdf95262700 1 Processor -- start 2024-03-12T03:15:09.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf95262700 1 -- start start 2024-03-12T03:15:09.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf95262700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf901a8e20 con 0x7fdf9010aa70 2024-03-12T03:15:09.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf95262700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf901aa040 con 0x7fdf90101de0 2024-03-12T03:15:09.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf95262700 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf901ab260 con 0x7fdf9010e610 2024-03-12T03:15:09.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf8f7fe700 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fdf9010e610 0x7fdf901a75a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:53828/0 (socket says 172.21.15.94:53828) 2024-03-12T03:15:09.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf8f7fe700 1 -- 172.21.15.94:0/3304698942 learned_addr learned my addr 172.21.15.94:0/3304698942 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:15:09.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf8e7fc700 1 --1- 172.21.15.94:0/3304698942 >> v1:172.21.15.94:6789/0 conn(0x7fdf9010aa70 0x7fdf901a3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:40814/0 (socket says 172.21.15.94:40814) 2024-03-12T03:15:09.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219534335 0 0) 0x7fdf901a8e20 con 0x7fdf9010aa70 2024-03-12T03:15:09.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf780031f0 con 0x7fdf9010aa70 2024-03-12T03:15:09.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567770490 0 0) 0x7fdf901ab260 con 0x7fdf9010e610 2024-03-12T03:15:09.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf901a8e20 con 0x7fdf9010e610 2024-03-12T03:15:09.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755491209 0 0) 0x7fdf901aa040 con 0x7fdf90101de0 2024-03-12T03:15:09.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf901ab260 con 0x7fdf90101de0 2024-03-12T03:15:09.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.608+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4286182210 0 0) 0x7fdf780031f0 con 0x7fdf9010aa70 2024-03-12T03:15:09.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf901aa040 con 0x7fdf9010aa70 2024-03-12T03:15:09.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fdf840042d0 con 0x7fdf9010aa70 2024-03-12T03:15:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3531857072 0 0) 0x7fdf901a8e20 con 0x7fdf9010e610 2024-03-12T03:15:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf780031f0 con 0x7fdf9010e610 2024-03-12T03:15:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fdf800013d0 con 0x7fdf9010e610 2024-03-12T03:15:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2130197379 0 0) 0x7fdf901ab260 con 0x7fdf90101de0 2024-03-12T03:15:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf901a8e20 con 0x7fdf90101de0 2024-03-12T03:15:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3066677522 0 0) 0x7fdf901aa040 con 0x7fdf9010aa70 2024-03-12T03:15:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 >> v1:172.21.15.94:6790/0 conn(0x7fdf9010e610 legacy=0x7fdf901a75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 >> v1:172.21.15.151:6789/0 conn(0x7fdf90101de0 legacy=0x7fdf90107070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf901ac4a0 con 0x7fdf9010aa70 2024-03-12T03:15:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf95262700 1 -- 172.21.15.94:0/3304698942 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf901ab490 con 0x7fdf9010aa70 2024-03-12T03:15:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf95262700 1 -- 172.21.15.94:0/3304698942 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf901ab950 con 0x7fdf9010aa70 2024-03-12T03:15:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fdf84001ab0 con 0x7fdf9010aa70 2024-03-12T03:15:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.609+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fdf84004ea0 con 0x7fdf9010aa70 2024-03-12T03:15:09.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.610+0000 7fdf95262700 1 -- 172.21.15.94:0/3304698942 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7c005120 con 0x7fdf9010aa70 2024-03-12T03:15:09.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.610+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7fdf84004e80 con 0x7fdf9010aa70 2024-03-12T03:15:09.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.611+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 285882637 0 0) 0x7fdf84089330 con 0x7fdf9010aa70 2024-03-12T03:15:09.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.615+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fdf84056b70 con 0x7fdf9010aa70 2024-03-12T03:15:09.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.853+0000 7fdf95262700 1 -- 172.21.15.94:0/3304698942 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdf7c005410 con 0x7fdf9010aa70 2024-03-12T03:15:09.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.854+0000 7fdf77fff700 1 -- 172.21.15.94:0/3304698942 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7fdf8405a5c0 con 0x7fdf9010aa70 2024-03-12T03:15:09.857 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-12T03:15:09.857 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-12T03:15:09.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.855+0000 7fdf95262700 1 -- 172.21.15.94:0/3304698942 >> v1:172.21.15.94:6800/2099808905 conn(0x7fdf7806ecf0 legacy=0x7fdf780711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:09.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.855+0000 7fdf95262700 1 -- 172.21.15.94:0/3304698942 >> v1:172.21.15.94:6789/0 conn(0x7fdf9010aa70 legacy=0x7fdf901a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:09.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.856+0000 7fdf95262700 1 -- 172.21.15.94:0/3304698942 shutdown_connections 2024-03-12T03:15:09.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.856+0000 7fdf95262700 1 -- 172.21.15.94:0/3304698942 >> 172.21.15.94:0/3304698942 conn(0x7fdf900fd690 msgr2=0x7fdf900fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:09.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.856+0000 7fdf95262700 1 -- 172.21.15.94:0/3304698942 shutdown_connections 2024-03-12T03:15:09.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-12T03:15:09.856+0000 7fdf95262700 1 -- 172.21.15.94:0/3304698942 wait complete. 2024-03-12T03:15:10.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3304698942' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:15:10.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3304698942' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:15:10.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3304698942' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:15:10.543 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-12T03:15:10.543 INFO:tasks.cephadm:Setup complete, yielding 2024-03-12T03:15:10.543 INFO:teuthology.run_tasks:Running task workunit... 2024-03-12T03:15:10.558 INFO:tasks.workunit:Pulling workunits from ref 1c349451176cc5b4ebfb24b22eaaa754e05cff6c 2024-03-12T03:15:10.558 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-12T03:15:10.558 DEBUG:teuthology.orchestra.run.smithi094:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-12T03:15:10.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:15:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-12T03:15:10.575 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-12T03:15:10.630 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-12T03:15:10.630 DEBUG:teuthology.orchestra.run.smithi094:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-12T03:15:10.685 INFO:tasks.workunit:timeout=3h 2024-03-12T03:15:10.685 INFO:tasks.workunit:cleanup=True 2024-03-12T03:15:10.685 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1c349451176cc5b4ebfb24b22eaaa754e05cff6c 2024-03-12T03:15:10.741 INFO:tasks.workunit.client.0.smithi094.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-12T03:15:10.913 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:15:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:15:11.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:10 smithi094 ceph-mon[29291]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:11.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:10 smithi094 ceph-mon[40484]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:11.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:10 smithi151 ceph-mon[32923]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:12.195 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:11 smithi151 systemd[1]: Starting Ceph grafana.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:15:12.446 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 podman[63513]: 2024-03-12 03:15:12.094992201 +0000 UTC m=+0.017200723 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-12T03:15:12.447 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 podman[63513]: 2024-03-12 03:15:12.199208378 +0000 UTC m=+0.121416933 container create 3bde7f9e682038bcf2c8bdcc9cacec45b44c066fef34063eee5d4afb390ee570 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, vendor=Red Hat, Inc., io.openshift.expose-services=, version=8.5, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, name=ubi8, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, distribution-scope=public, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664) 2024-03-12T03:15:12.447 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 podman[63513]: 2024-03-12 03:15:12.385142264 +0000 UTC m=+0.307350787 container init 3bde7f9e682038bcf2c8bdcc9cacec45b44c066fef34063eee5d4afb390ee570 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a, version=8.5, io.buildah.version=1.32.0, io.openshift.expose-services=, vcs-type=git, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, build-date=2022-04-29T11:48:11.055897) 2024-03-12T03:15:12.447 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 podman[63513]: 2024-03-12 03:15:12.387756431 +0000 UTC m=+0.309964941 container start 3bde7f9e682038bcf2c8bdcc9cacec45b44c066fef34063eee5d4afb390ee570 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., distribution-scope=public, io.buildah.version=1.32.0, release=239.1651231664, architecture=x86_64, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git) 2024-03-12T03:15:12.447 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 bash[63513]: 3bde7f9e682038bcf2c8bdcc9cacec45b44c066fef34063eee5d4afb390ee570 2024-03-12T03:15:12.698 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 systemd[1]: Started Ceph grafana.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:15:12.698 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.541903109Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-12T03:15:12.699 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.542093294Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-12T03:15:12.699 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.542113157Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-12T03:15:12.699 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.542120392Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-12T03:15:12.699 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.542126803Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-12T03:15:12.699 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.542132816Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-12T03:15:12.699 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.542139288Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-12T03:15:12.699 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.542145483Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-12T03:15:12.699 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.542151368Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-12T03:15:12.699 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.542157114Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-12T03:15:12.699 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.542162962Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-12T03:15:12.699 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.542168443Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-12T03:15:12.700 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.542174144Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-12T03:15:12.700 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=settings t=2024-03-12T03:15:12.542179886Z level=info msg="App mode production" 2024-03-12T03:15:12.700 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=sqlstore t=2024-03-12T03:15:12.542223514Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-12T03:15:12.700 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=sqlstore t=2024-03-12T03:15:12.542240022Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-12T03:15:12.700 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.543081074Z level=info msg="Starting DB migrations" 2024-03-12T03:15:12.700 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.54317885Z level=info msg="Executing migration" id="create migration_log table" 2024-03-12T03:15:12.700 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.543696467Z level=info msg="Executing migration" id="create user table" 2024-03-12T03:15:12.700 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.544148044Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-12T03:15:12.700 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.546591178Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-12T03:15:12.700 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.54705106Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-12T03:15:12.700 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.549799799Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-12T03:15:12.701 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.552243285Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-12T03:15:12.701 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.553127089Z level=info msg="Executing migration" id="create user table v2" 2024-03-12T03:15:12.701 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.555661594Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-12T03:15:12.701 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.558230333Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-12T03:15:12.701 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.560953739Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-12T03:15:12.701 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.56135893Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-12T03:15:12.701 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.563852073Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-12T03:15:12.701 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.564312208Z level=info msg="Executing migration" id="Update user table charset" 2024-03-12T03:15:12.701 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.566439345Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-12T03:15:12.701 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.567244978Z level=info msg="Executing migration" id="Add missing user data" 2024-03-12T03:15:12.701 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.56818415Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-12T03:15:12.702 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.568843348Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-12T03:15:12.702 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.569723547Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-12T03:15:12.702 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.570487191Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-12T03:15:12.702 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.571663824Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-12T03:15:12.703 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.572649807Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-12T03:15:12.703 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.573242764Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-12T03:15:12.703 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.573637832Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-12T03:15:12.703 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.574477023Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-12T03:15:12.703 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.575398016Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-12T03:15:12.703 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.576068174Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-12T03:15:12.703 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.577050495Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-12T03:15:12.704 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.577738447Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-12T03:15:12.704 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.57808895Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-12T03:15:12.704 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.57852384Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-12T03:15:12.704 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.579120114Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-12T03:15:12.704 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.579514174Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-12T03:15:12.704 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.579951375Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-12T03:15:12.704 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.580342769Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-12T03:15:12.704 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.5808132Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-12T03:15:12.704 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.581218209Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-12T03:15:12.704 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.582165241Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-12T03:15:12.704 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.582935469Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-12T03:15:12.704 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.583637191Z level=info msg="Executing migration" id="create star table" 2024-03-12T03:15:12.704 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.584452461Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-12T03:15:12.705 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.584908956Z level=info msg="Executing migration" id="create org table v1" 2024-03-12T03:15:12.705 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.585323449Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-12T03:15:12.705 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.586268465Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-12T03:15:12.705 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.586910092Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-12T03:15:12.705 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.58791508Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-12T03:15:12.705 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.588558629Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-12T03:15:12.705 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.589499992Z level=info msg="Executing migration" id="Update org table charset" 2024-03-12T03:15:12.705 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.59016655Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-12T03:15:12.705 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.593006465Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-12T03:15:12.705 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.596534287Z level=info msg="Executing migration" id="create dashboard table" 2024-03-12T03:15:12.705 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.599270726Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-12T03:15:12.705 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.601961226Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-12T03:15:12.706 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.604836222Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-12T03:15:12.706 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.607856893Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-12T03:15:12.706 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.610422412Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-12T03:15:12.706 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.613203783Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-12T03:15:12.706 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.615729831Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-12T03:15:12.706 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.618496733Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-12T03:15:12.706 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.621272883Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-12T03:15:12.706 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.623968409Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-12T03:15:12.706 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.627734944Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-12T03:15:12.706 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.630618558Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-12T03:15:12.706 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.633220585Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-12T03:15:12.707 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.635891716Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-12T03:15:12.707 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.638712625Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-12T03:15:12.707 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.641570045Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-12T03:15:12.707 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.644253962Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-12T03:15:12.707 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.64684419Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-12T03:15:12.707 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.649471422Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-12T03:15:12.707 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.652048213Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-12T03:15:12.707 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.654785758Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-12T03:15:12.707 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.657438668Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-12T03:15:12.707 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.660072872Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-12T03:15:12.707 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.662757335Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-12T03:15:12.707 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.665254646Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-12T03:15:12.708 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.667931272Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-12T03:15:12.708 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.67052701Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-12T03:15:12.708 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.673117927Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-12T03:15:12.708 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.675592113Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-12T03:15:12.708 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.678300921Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-12T03:15:12.708 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.680757962Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-12T03:15:12.708 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.683365822Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-12T03:15:12.708 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.68589275Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-12T03:15:12.708 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.686737945Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-12T03:15:12.708 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.687530631Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-12T03:15:12.708 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.68833224Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-12T03:15:12.708 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.689121369Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-12T03:15:12.709 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.690061996Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-12T03:15:12.709 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.69085048Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-12T03:15:12.709 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.691588875Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-12T03:15:12.709 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.692510542Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-12T03:15:12.709 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.693155855Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-12T03:15:12.709 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.694118524Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-12T03:15:12.709 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.694786148Z level=info msg="Executing migration" id="create data_source table" 2024-03-12T03:15:12.709 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.695623399Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-12T03:15:12.709 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.696403201Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-12T03:15:12.709 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.697193349Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-12T03:15:12.709 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.697945259Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-12T03:15:12.710 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.698724744Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-12T03:15:12.950 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.700343131Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-12T03:15:12.951 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.701107616Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-12T03:15:12.951 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.701995383Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-12T03:15:12.951 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.70282199Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-12T03:15:12.951 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.703634736Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-12T03:15:12.951 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.704458558Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-12T03:15:12.951 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.705311439Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-12T03:15:12.951 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.706131003Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-12T03:15:12.951 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.708432983Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-12T03:15:12.952 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.711294852Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-12T03:15:12.952 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.71407225Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-12T03:15:12.952 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.716817931Z level=info msg="Executing migration" id="Add uid column" 2024-03-12T03:15:12.952 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.719542641Z level=info msg="Executing migration" id="Update uid value" 2024-03-12T03:15:12.952 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.72243994Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-12T03:15:12.952 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.725044128Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-12T03:15:12.952 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.727893622Z level=info msg="Executing migration" id="create api_key table" 2024-03-12T03:15:12.952 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.730804743Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-12T03:15:12.952 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.733243269Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-12T03:15:12.952 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.736248649Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-12T03:15:12.954 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.738938262Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-12T03:15:12.954 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.741693348Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-12T03:15:12.954 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.744418451Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-12T03:15:12.954 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.747263902Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-12T03:15:12.955 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.748407019Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-12T03:15:12.955 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.749280089Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-12T03:15:12.955 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.750121603Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-12T03:15:12.955 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.751007456Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-12T03:15:12.955 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.751877949Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-12T03:15:12.955 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.752678707Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-12T03:15:12.955 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.753060122Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-12T03:15:12.955 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.754005027Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-12T03:15:12.955 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.754623565Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-12T03:15:12.955 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.755469945Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-12T03:15:12.955 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.756398317Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.757243314Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.757792723Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.758772Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.759500804Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.760445388Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.760994692Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.761872208Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.762932355Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.763571174Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.763861151Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.764467974Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.765033193Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.765906987Z level=info msg="Executing migration" id="create quota table v1" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.766973612Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-12T03:15:12.956 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.767646167Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.767916747Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.76847076Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.769411311Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.770212003Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.770505185Z level=info msg="Executing migration" id="create session table" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.770946077Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.771735903Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.772073673Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.77248257Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.773735561Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.774162959Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.774463488Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.774742579Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-12T03:15:12.957 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.77503832Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-12T03:15:12.958 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.7754391Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-12T03:15:12.958 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.77571088Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-12T03:15:12.958 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.77632512Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-12T03:15:12.958 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.777144193Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-12T03:15:12.958 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.777776206Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-12T03:15:12.958 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.778407721Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-12T03:15:12.958 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.781120952Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-12T03:15:12.958 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.781581513Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-12T03:15:12.958 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.784423508Z level=info msg="Executing migration" id="create alert table v1" 2024-03-12T03:15:12.958 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.784891011Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-12T03:15:12.958 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.787845065Z level=info msg="Executing migration" id="add index alert state" 2024-03-12T03:15:12.958 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.790679395Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-12T03:15:12.958 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.791140954Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.791541826Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.79195157Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.79238405Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.79392828Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.794317304Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.79473554Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.795029717Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.795384123Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.795733656Z level=info msg="Executing migration" id="Add column is_default" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.796358347Z level=info msg="Executing migration" id="Add column frequency" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.796963611Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.797578959Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.798209029Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.79862568Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-12T03:15:12.959 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.798888797Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.799181515Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.799559621Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.799975876Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.800352364Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.80071565Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.801161332Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.801780052Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.802441612Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.802759396Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.803188545Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.803579705Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.804212154Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.804509284Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.804891635Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-12T03:15:12.960 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.805313075Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.805625609Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.806066873Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.806698125Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.808953376Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.809434432Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.809898866Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.812948364Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.81329356Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.81394155Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.814358264Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.817612855Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.820456847Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.820901021Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-12T03:15:12.961 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.823872113Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.826789194Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.829696287Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.830303086Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.833127368Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.835945846Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.838736382Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.839524478Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.842335802Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.845009046Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.847927844Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.848296426Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.851198644Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.853875795Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-12T03:15:12.962 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.856551283Z level=info msg="Executing migration" id="Move region to single row" 2024-03-12T03:15:12.963 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.856956353Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-12T03:15:12.963 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.859832451Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-12T03:15:12.963 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.862466991Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-12T03:15:12.963 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.865396485Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-12T03:15:12.963 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.865824529Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-12T03:15:12.963 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.868661811Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-12T03:15:12.963 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.871429001Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-12T03:15:12.963 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.874240157Z level=info msg="Executing migration" id="create test_data table" 2024-03-12T03:15:12.963 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.874812293Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-12T03:15:12.963 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.875219655Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-12T03:15:12.963 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.876207199Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-12T03:15:12.963 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.876911696Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-12T03:15:12.963 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.877260243Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-12T03:15:12.964 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.878096628Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-12T03:15:12.964 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.879054344Z level=info msg="Executing migration" id="create team table" 2024-03-12T03:15:12.964 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.879734364Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-12T03:15:12.964 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.880187012Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-12T03:15:12.964 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.881169417Z level=info msg="Executing migration" id="create team member table" 2024-03-12T03:15:12.964 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.882000687Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-12T03:15:12.964 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.882710922Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-12T03:15:12.964 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.883133698Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-12T03:15:12.964 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.883962781Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-12T03:15:12.964 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.884780459Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-12T03:15:12.964 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.885595964Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-12T03:15:12.964 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.886313841Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-12T03:15:12.964 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.88921202Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.892217808Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.895099394Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.895608686Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.898617461Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.90144922Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.904218089Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.906718531Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.909337215Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.912169344Z level=info msg="Executing migration" id="create tag table" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.914762141Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.917396677Z level=info msg="Executing migration" id="create login attempt table" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.919956026Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.922585279Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.925174185Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-12T03:15:12.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.927807457Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-12T03:15:12.966 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.930653243Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-12T03:15:12.966 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.933255149Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-12T03:15:12.966 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.936046823Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-12T03:15:12.966 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.938520087Z level=info msg="Executing migration" id="create user auth table" 2024-03-12T03:15:12.966 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.941129189Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-12T03:15:12.966 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.943617614Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-12T03:15:12.966 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.946307327Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-12T03:15:12.966 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.948759435Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-12T03:15:12.966 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.951357142Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-12T03:15:13.202 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.953932798Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-12T03:15:13.203 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.956455537Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-12T03:15:13.203 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.959156989Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-12T03:15:13.203 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.96168698Z level=info msg="Executing migration" id="create server_lock table" 2024-03-12T03:15:13.203 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.964233829Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-12T03:15:13.203 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.966747132Z level=info msg="Executing migration" id="create user auth token table" 2024-03-12T03:15:13.203 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.969299115Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-12T03:15:13.203 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.971882853Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-12T03:15:13.203 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.974463543Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-12T03:15:13.203 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.976915181Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-12T03:15:13.203 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.977766177Z level=info msg="Executing migration" id="create cache_data table" 2024-03-12T03:15:13.203 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.978566356Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-12T03:15:13.203 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.979356377Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.980171731Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.981048914Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.981939229Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.982629835Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.983438357Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.984357075Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.985074413Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.985999959Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.986734759Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.987522195Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.988358047Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.98927821Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.990255227Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-12T03:15:13.204 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.991056389Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-12T03:15:13.205 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.992116498Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-12T03:15:13.205 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.992851819Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-12T03:15:13.205 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.993766092Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-12T03:15:13.205 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.994444281Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-12T03:15:13.205 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.995352018Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-12T03:15:13.205 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.996110842Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-12T03:15:13.205 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.997047613Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-12T03:15:13.205 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.997726517Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-12T03:15:13.205 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.998676904Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-12T03:15:13.207 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:12 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:12.999614631Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-12T03:15:13.207 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.000424776Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-12T03:15:13.207 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.001236123Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-12T03:15:13.207 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.004748233Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-12T03:15:13.207 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.008244846Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-12T03:15:13.207 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.009045136Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-12T03:15:13.207 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.009974381Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-12T03:15:13.207 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.011006789Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-12T03:15:13.207 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.011783419Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-12T03:15:13.207 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.014735981Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-12T03:15:13.207 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.017522247Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-12T03:15:13.207 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.019990113Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.022632222Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.025549407Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.028044903Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.030801518Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.033294446Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.033749925Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.036721518Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.039402972Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.042173194Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.043161357Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.045957641Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.048727901Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.051267385Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.051735382Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-12T03:15:13.208 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.054496803Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.057195237Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.059747769Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.060611223Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.06310535Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.065909134Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.068490783Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.068943554Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.071710213Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.074153628Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.07658058Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.077565943Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.080153759Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.082766324Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-12T03:15:13.209 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.08544775Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.08594765Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.088508084Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.091109427Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.091941637Z level=info msg="Executing migration" id="create alert_image table" 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.092615256Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.093412947Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.094310521Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.095026837Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.095511218Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.095597388Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.096271845Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.09666827Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.09709528Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-12T03:15:13.210 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.09722614Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.097340096Z level=info msg="alerts found to migrate" alerts=0 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.097784365Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.098258597Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.099097809Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.100085442Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.100906667Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.101624515Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.101912798Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.102323832Z level=info msg="Executing migration" id="create data_keys table" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.10280999Z level=info msg="Executing migration" id="create secrets table" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.10381169Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.108175102Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.109081255Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.109913885Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.114149418Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-12T03:15:13.211 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.118367922Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-12T03:15:13.212 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.118954187Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-12T03:15:13.212 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.119953843Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-12T03:15:13.212 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.120705114Z level=info msg="Executing migration" id="create permission table" 2024-03-12T03:15:13.212 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.121575847Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-12T03:15:13.212 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.122091591Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-12T03:15:13.212 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.12293668Z level=info msg="Executing migration" id="create role table" 2024-03-12T03:15:13.212 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.123892084Z level=info msg="Executing migration" id="add column display_name" 2024-03-12T03:15:13.212 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.12488427Z level=info msg="Executing migration" id="add column group_name" 2024-03-12T03:15:13.212 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.125819244Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-12T03:15:13.212 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.126707911Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-12T03:15:13.212 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.127636628Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-12T03:15:13.212 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.128445581Z level=info msg="Executing migration" id="create team role table" 2024-03-12T03:15:13.212 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.128860724Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.129740493Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.13074217Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.131436847Z level=info msg="Executing migration" id="create user role table" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.131868666Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.134780348Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.137541917Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.14021174Z level=info msg="Executing migration" id="create builtin role table" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.140650881Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.143648391Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.146407168Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.147430012Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.149370369Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.152220513Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.152654149Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-12T03:15:13.213 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.153061507Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.153468689Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.153912304Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.154898698Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.155313111Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.155765993Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.156037706Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.156465198Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.156878625Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.157286682Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.157591193Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.157915624Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.158317453Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.158771444Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-12T03:15:13.214 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.159765507Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.16007148Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.160505782Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.160945876Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.161412905Z level=info msg="Executing migration" id="add correlation config column" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.162404028Z level=info msg="Executing migration" id="create entity_events table" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.162844236Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.163258809Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.163343781Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.16362407Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.163700126Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.16394894Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.164334072Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.164737313Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.165190402Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T03:15:13.215 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.165648074Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.166032964Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.166470126Z level=info msg="Executing migration" id="Drop public config table" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.166825362Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.167265323Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.167689044Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.168116611Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.168550461Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.171538382Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.172574086Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.173628875Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.173943382Z level=info msg="Executing migration" id="add share column" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.174969749Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.175406794Z level=info msg="Executing migration" id="create file table" 2024-03-12T03:15:13.216 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.175830483Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.178608678Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.179135654Z level=info msg="Executing migration" id="create file_meta table" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.182254549Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.18276928Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.185741975Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.186136497Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.188694874Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.191490675Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.1925281Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.195399113Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.198093413Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.200650648Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.201068958Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-12T03:15:13.217 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.203811587Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-12T03:15:13.465 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.206586119Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-12T03:15:13.465 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.209357734Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-12T03:15:13.465 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.20994403Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-12T03:15:13.465 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.222994673Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-12T03:15:13.465 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.225824271Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-12T03:15:13.465 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.228343015Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-12T03:15:13.465 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.231764311Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-12T03:15:13.465 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.23458987Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-12T03:15:13.466 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.237213106Z level=info msg="Executing migration" id="create folder table" 2024-03-12T03:15:13.466 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.239990654Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-12T03:15:13.466 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.240527187Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-12T03:15:13.466 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.241536872Z level=info msg="Executing migration" id="Update folder title length" 2024-03-12T03:15:13.466 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.242244569Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-12T03:15:13.466 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=migrator t=2024-03-12T03:15:13.24321035Z level=info msg="migrations completed" performed=485 skipped=0 duration=700.050817ms 2024-03-12T03:15:13.466 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=sqlstore t=2024-03-12T03:15:13.243822148Z level=info msg="Created default organization" 2024-03-12T03:15:13.466 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=plugin.loader t=2024-03-12T03:15:13.300778025Z level=info msg="Plugin registered" pluginID=input 2024-03-12T03:15:13.466 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=plugin.loader t=2024-03-12T03:15:13.327859386Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-12T03:15:13.466 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=plugin.loader t=2024-03-12T03:15:13.32788454Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-12T03:15:13.466 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=secrets t=2024-03-12T03:15:13.327927322Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-12T03:15:13.469 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=query_data t=2024-03-12T03:15:13.330071401Z level=info msg="Query Service initialization" 2024-03-12T03:15:13.469 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=live.push_http t=2024-03-12T03:15:13.332458296Z level=info msg="Live Push Gateway initialization" 2024-03-12T03:15:13.772 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:13.772 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:13.772 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:13.772 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:13.772 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-mon[32923]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:13.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:13 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:13.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:13 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:13.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:13 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:13.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:13 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:13.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:13 smithi094 ceph-mon[29291]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:13.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:13 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:13.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:13 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:13.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:13 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:13.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:13 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:13.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:13 smithi094 ceph-mon[40484]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:13.961 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:13 smithi094 bash[76358]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-12T03:15:14.035 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=infra.usagestats.collector t=2024-03-12T03:15:13.988240763Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-12T03:15:14.035 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=provisioning.datasources t=2024-03-12T03:15:13.98958859Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-12T03:15:14.035 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=provisioning.datasources t=2024-03-12T03:15:14.000699167Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-12T03:15:14.035 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=provisioning.alerting t=2024-03-12T03:15:14.01189888Z level=info msg="starting to provision alerting" 2024-03-12T03:15:14.035 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=provisioning.alerting t=2024-03-12T03:15:14.01192608Z level=info msg="finished to provision alerting" 2024-03-12T03:15:14.035 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=grafanaStorageLogger t=2024-03-12T03:15:14.01209153Z level=info msg="storage starting" 2024-03-12T03:15:14.035 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=ngalert.state.manager t=2024-03-12T03:15:14.012192205Z level=info msg="Warming state cache for startup" 2024-03-12T03:15:14.035 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=ngalert.state.manager t=2024-03-12T03:15:14.012344379Z level=info msg="State cache has been initialized" states=0 duration=151.502µs 2024-03-12T03:15:14.035 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=ticker t=2024-03-12T03:15:14.012483482Z level=info msg=starting first_tick=2024-03-12T03:15:20Z 2024-03-12T03:15:14.035 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=ngalert.multiorg.alertmanager t=2024-03-12T03:15:14.012556712Z level=info msg="starting MultiOrg Alertmanager" 2024-03-12T03:15:14.035 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=http.server t=2024-03-12T03:15:14.014127697Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-12T03:15:14.036 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:15:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=sqlstore.transactions t=2024-03-12T03:15:14.033093927Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-12T03:15:14.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:14 smithi094 ceph-mon[29291]: Deploying daemon node-exporter.a on smithi094 2024-03-12T03:15:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:14 smithi094 ceph-mon[40484]: Deploying daemon node-exporter.a on smithi094 2024-03-12T03:15:14.960 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:14 smithi094 bash[76358]: Getting image source signatures 2024-03-12T03:15:14.960 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:14 smithi094 bash[76358]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-12T03:15:14.960 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:14 smithi094 bash[76358]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-12T03:15:14.960 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:14 smithi094 bash[76358]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-12T03:15:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:14 smithi151 ceph-mon[32923]: Deploying daemon node-exporter.a on smithi094 2024-03-12T03:15:15.800 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:15 smithi094 ceph-mon[29291]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:15.800 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:15 smithi094 ceph-mon[40484]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:15 smithi151 ceph-mon[32923]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:16.459 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 bash[76358]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-12T03:15:16.459 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 bash[76358]: Writing manifest to image destination 2024-03-12T03:15:16.818 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-mon[40484]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:16.819 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-mon[29291]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:16.819 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 podman[76358]: 2024-03-12 03:15:16.575671785 +0000 UTC m=+2.926568209 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-12T03:15:16.819 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 podman[76358]: 2024-03-12 03:15:16.707818228 +0000 UTC m=+3.058714637 container create f3063f7b48c78f1a1d78eb226f23a0a0884adcbe7f4495e91b91da3c08c1c65d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:16 smithi151 ceph-mon[32923]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:17.210 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 podman[76358]: 2024-03-12 03:15:16.934944207 +0000 UTC m=+3.285840634 container init f3063f7b48c78f1a1d78eb226f23a0a0884adcbe7f4495e91b91da3c08c1c65d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:17.210 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 podman[76358]: 2024-03-12 03:15:16.937688612 +0000 UTC m=+3.288585026 container start f3063f7b48c78f1a1d78eb226f23a0a0884adcbe7f4495e91b91da3c08c1c65d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.942Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.942Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.943Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.943Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.943Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.943Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=arp 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-12T03:15:17.211 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-12T03:15:17.212 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-12T03:15:17.212 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-12T03:15:17.212 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=edac 2024-03-12T03:15:17.212 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-12T03:15:17.212 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-12T03:15:17.212 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=os 2024-03-12T03:15:17.213 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-12T03:15:17.214 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-12T03:15:17.214 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-12T03:15:17.214 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-12T03:15:17.214 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-12T03:15:17.214 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-12T03:15:17.214 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-12T03:15:17.214 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=stat 2024-03-12T03:15:17.214 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-12T03:15:17.214 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-12T03:15:17.214 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-12T03:15:17.214 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=time 2024-03-12T03:15:17.214 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-12T03:15:17.214 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=uname 2024-03-12T03:15:17.215 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.944Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-12T03:15:17.215 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.945Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-12T03:15:17.215 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.945Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-12T03:15:17.215 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.945Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-12T03:15:17.215 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a[76421]: ts=2024-03-12T03:15:16.945Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-12T03:15:17.215 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:17 smithi094 bash[76358]: f3063f7b48c78f1a1d78eb226f23a0a0884adcbe7f4495e91b91da3c08c1c65d 2024-03-12T03:15:17.215 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:15:17 smithi094 systemd[1]: Started Ceph node-exporter.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:15:17.886 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:17.886 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:15:17.886 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:17.886 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:17.886 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:17.886 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:17 smithi151 ceph-mon[32923]: Deploying daemon node-exporter.b on smithi151 2024-03-12T03:15:17.886 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:15:17 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:15:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:15:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:17 smithi094 ceph-mon[40484]: Deploying daemon node-exporter.b on smithi151 2024-03-12T03:15:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:15:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:17.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:17.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:17 smithi094 ceph-mon[29291]: Deploying daemon node-exporter.b on smithi151 2024-03-12T03:15:18.214 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:18 smithi151 systemd[1]: Starting Ceph node-exporter.b for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:15:18.605 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:18 smithi151 bash[63792]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-12T03:15:18.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:18 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:18 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:18 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:18.964 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:18 smithi151 bash[63792]: Getting image source signatures 2024-03-12T03:15:18.964 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:18 smithi151 bash[63792]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-12T03:15:18.965 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:18 smithi151 bash[63792]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-12T03:15:18.965 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:18 smithi151 bash[63792]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-12T03:15:19.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:19 smithi094 ceph-mon[29291]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:19 smithi094 ceph-mon[40484]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:19 smithi151 ceph-mon[32923]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:20.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:15:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:15:21.609 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:21 smithi151 bash[63792]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-12T03:15:21.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:21 smithi094 ceph-mon[40484]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:21 smithi094 ceph-mon[29291]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:21 smithi151 ceph-mon[32923]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:23.464 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:23 smithi151 bash[63792]: Writing manifest to image destination 2024-03-12T03:15:23.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:23 smithi094 ceph-mon[40484]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:23.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:23 smithi094 ceph-mon[29291]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:23.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:23 smithi151 ceph-mon[32923]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:25.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:25 smithi094 ceph-mon[29291]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:25.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:25 smithi094 ceph-mon[40484]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:25.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:25 smithi151 ceph-mon[32923]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:25.964 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:25 smithi151 podman[63792]: 2024-03-12 03:15:25.80489223 +0000 UTC m=+7.634358866 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-12T03:15:27.164 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:26 smithi151 podman[63792]: 2024-03-12 03:15:26.910686897 +0000 UTC m=+8.740153515 container create a6926598bdcce3ec6dea19d1bb8b8c92d760615572f91593da30bc72d1d628c6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:15:27.164 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 podman[63792]: 2024-03-12 03:15:27.155716292 +0000 UTC m=+8.985182910 container init a6926598bdcce3ec6dea19d1bb8b8c92d760615572f91593da30bc72d1d628c6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:15:27.164 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 podman[63792]: 2024-03-12 03:15:27.158543043 +0000 UTC m=+8.988009664 container start a6926598bdcce3ec6dea19d1bb8b8c92d760615572f91593da30bc72d1d628c6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:15:27.465 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.162Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-12T03:15:27.466 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.162Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-12T03:15:27.466 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.163Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-12T03:15:27.466 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.163Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T03:15:27.466 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.163Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-12T03:15:27.466 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-12T03:15:27.466 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-12T03:15:27.466 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=arp 2024-03-12T03:15:27.466 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-12T03:15:27.466 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-12T03:15:27.467 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-12T03:15:27.467 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-12T03:15:27.467 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-12T03:15:27.467 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-12T03:15:27.467 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-12T03:15:27.467 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-12T03:15:27.467 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=edac 2024-03-12T03:15:27.467 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-12T03:15:27.467 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-12T03:15:27.467 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-12T03:15:27.467 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-12T03:15:27.467 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-12T03:15:27.467 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-12T03:15:27.468 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-12T03:15:27.468 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-12T03:15:27.468 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-12T03:15:27.468 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-12T03:15:27.468 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-12T03:15:27.468 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-12T03:15:27.468 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-12T03:15:27.468 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-12T03:15:27.468 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-12T03:15:27.468 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-12T03:15:27.468 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=os 2024-03-12T03:15:27.468 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-12T03:15:27.468 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-12T03:15:27.469 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-12T03:15:27.469 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-12T03:15:27.469 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-12T03:15:27.469 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.164Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-12T03:15:27.469 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.165Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-12T03:15:27.469 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.165Z caller=node_exporter.go:117 level=info collector=stat 2024-03-12T03:15:27.469 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.165Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-12T03:15:27.469 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.165Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-12T03:15:27.469 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.165Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-12T03:15:27.469 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.165Z caller=node_exporter.go:117 level=info collector=time 2024-03-12T03:15:27.469 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.165Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-12T03:15:27.469 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.165Z caller=node_exporter.go:117 level=info collector=uname 2024-03-12T03:15:27.469 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.165Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-12T03:15:27.470 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.165Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-12T03:15:27.470 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.165Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-12T03:15:27.470 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.165Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-12T03:15:27.470 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b[63841]: ts=2024-03-12T03:15:27.165Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-12T03:15:27.470 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 bash[63792]: a6926598bdcce3ec6dea19d1bb8b8c92d760615572f91593da30bc72d1d628c6 2024-03-12T03:15:27.470 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:15:27 smithi151 systemd[1]: Started Ceph node-exporter.b for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:15:27.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:15:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:15:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[29291]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:15:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[40484]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:28 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:15:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:28 smithi151 ceph-mon[32923]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:28 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:28 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:28 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:28 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:28 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:28 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:15:29.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:30.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:30 smithi094 ceph-mon[29291]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:30 smithi094 ceph-mon[40484]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:30.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:15:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:15:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:30 smithi151 ceph-mon[32923]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:31.673 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:31 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:31.674 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:31 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:31.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:31 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:31.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:31 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:31 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:31 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:32.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[29291]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:15:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:15:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:15:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:32.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[40484]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:32.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:15:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:15:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:15:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:32 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:32 smithi151 ceph-mon[32923]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:32 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:32 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:15:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:32 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:32 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:32.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:32 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:15:32.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:32 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:15:32.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:32 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:34.793 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:34 smithi094 ceph-mon[40484]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:34.794 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:34 smithi094 ceph-mon[40484]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-12T03:15:34.794 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:34 smithi094 ceph-mon[40484]: Reconfiguring daemon alertmanager.a on smithi094 2024-03-12T03:15:34.794 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:34 smithi094 ceph-mon[29291]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:34.794 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:34 smithi094 ceph-mon[29291]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-12T03:15:34.794 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:34 smithi094 ceph-mon[29291]: Reconfiguring daemon alertmanager.a on smithi094 2024-03-12T03:15:34.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:34 smithi151 ceph-mon[32923]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:34.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:34 smithi151 ceph-mon[32923]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-12T03:15:34.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:34 smithi151 ceph-mon[32923]: Reconfiguring daemon alertmanager.a on smithi094 2024-03-12T03:15:36.209 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:36 smithi094 systemd[1]: Stopping Ceph alertmanager.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:15:36.689 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:36 smithi094 ceph-mon[40484]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:36.689 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:36 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[75451]: ts=2024-03-12T03:15:36.327Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:15:36.690 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:36 smithi094 podman[77232]: 2024-03-12 03:15:36.378735077 +0000 UTC m=+0.183837834 container died edda9a59ed7ec8e9a4393da5d25fb97efca0dfc285a9664e6edc6918b90edd39 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:36.690 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:36 smithi094 ceph-mon[29291]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:36.959 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:36 smithi094 podman[77232]: 2024-03-12 03:15:36.82046713 +0000 UTC m=+0.625569897 container cleanup edda9a59ed7ec8e9a4393da5d25fb97efca0dfc285a9664e6edc6918b90edd39 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:36.959 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:36 smithi094 bash[77232]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a 2024-03-12T03:15:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:36 smithi151 ceph-mon[32923]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:37.459 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:37 smithi094 podman[77247]: 2024-03-12 03:15:37.105429828 +0000 UTC m=+0.731620796 container remove edda9a59ed7ec8e9a4393da5d25fb97efca0dfc285a9664e6edc6918b90edd39 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:37.460 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:37 smithi094 podman[77247]: 2024-03-12 03:15:37.197867791 +0000 UTC m=+0.824058761 volume remove afd6343a3874d8b31b4fbb1b438839faaa534248ea86ef1e476ac9875cd6e305 2024-03-12T03:15:37.759 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:37 smithi094 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-12T03:15:37.759 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:37 smithi094 systemd[1]: Stopped Ceph alertmanager.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:15:37.759 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:37 smithi094 systemd[1]: Starting Ceph alertmanager.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:15:37.759 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:37 smithi094 podman[77338]: 2024-03-12 03:15:37.627023406 +0000 UTC m=+0.018490571 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-12T03:15:37.759 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:37 smithi094 podman[77338]: 2024-03-12 03:15:37.757072166 +0000 UTC m=+0.148539320 volume create c83964b83dc7bb90ab63cc95e5747f5b3c0608f16749531d06ac1e248d19f983 2024-03-12T03:15:37.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:15:37 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:15:38.090 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:37 smithi094 podman[77338]: 2024-03-12 03:15:37.874596736 +0000 UTC m=+0.266063896 container create e3549d8f9563f70d4c72f1abec3831ab16f696586d0f814b6c7bd7f0f31402f0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:38.460 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:38 smithi094 podman[77338]: 2024-03-12 03:15:38.211254465 +0000 UTC m=+0.602721632 container init e3549d8f9563f70d4c72f1abec3831ab16f696586d0f814b6c7bd7f0f31402f0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:38.460 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:38 smithi094 podman[77338]: 2024-03-12 03:15:38.21386424 +0000 UTC m=+0.605331389 container start e3549d8f9563f70d4c72f1abec3831ab16f696586d0f814b6c7bd7f0f31402f0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:38.460 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[77353]: ts=2024-03-12T03:15:38.225Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-12T03:15:38.460 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[77353]: ts=2024-03-12T03:15:38.225Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-12T03:15:38.460 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[77353]: ts=2024-03-12T03:15:38.226Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.94 port=9094 2024-03-12T03:15:38.460 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[77353]: ts=2024-03-12T03:15:38.227Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-12T03:15:38.461 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[77353]: ts=2024-03-12T03:15:38.255Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:15:38.461 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[77353]: ts=2024-03-12T03:15:38.256Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:15:38.461 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[77353]: ts=2024-03-12T03:15:38.260Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-12T03:15:38.461 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[77353]: ts=2024-03-12T03:15:38.260Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-12T03:15:38.461 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:38 smithi094 bash[77338]: e3549d8f9563f70d4c72f1abec3831ab16f696586d0f814b6c7bd7f0f31402f0 2024-03-12T03:15:38.461 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:38 smithi094 systemd[1]: Started Ceph alertmanager.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:15:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-mon[29291]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-mon[40484]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:38 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:38 smithi151 ceph-mon[32923]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:38 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:38 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:39.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:39.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:39 smithi094 ceph-mon[29291]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:15:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:39 smithi094 ceph-mon[40484]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:15:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:39 smithi151 ceph-mon[32923]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:15:40.531 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[77353]: ts=2024-03-12T03:15:40.228Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000774523s 2024-03-12T03:15:40.586 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:40 smithi151 ceph-mon[32923]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:40.586 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:40 smithi151 ceph-mon[32923]: Reconfiguring daemon prometheus.a on smithi151 2024-03-12T03:15:40.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:40 smithi094 ceph-mon[29291]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:40.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:40 smithi094 ceph-mon[29291]: Reconfiguring daemon prometheus.a on smithi151 2024-03-12T03:15:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:40 smithi094 ceph-mon[40484]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:40 smithi094 ceph-mon[40484]: Reconfiguring daemon prometheus.a on smithi151 2024-03-12T03:15:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:15:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:15:41.629 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:41 smithi151 ceph-mon[32923]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:41.950 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 systemd[1]: Stopping Ceph prometheus.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:15:41.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:41 smithi094 ceph-mon[29291]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:41.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:41 smithi094 ceph-mon[40484]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:42.209 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:15:41.948Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:15:42.210 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:15:41.948Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-12T03:15:42.210 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:15:41.948Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-12T03:15:42.210 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:15:41.948Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-12T03:15:42.210 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:15:41.948Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-12T03:15:42.210 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:15:41.948Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-12T03:15:42.210 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:15:41.948Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-12T03:15:42.210 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:15:41.948Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-12T03:15:42.210 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:15:41.949Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-12T03:15:42.210 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:15:41.949Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-12T03:15:42.210 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:15:41.949Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-12T03:15:42.210 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[61595]: ts=2024-03-12T03:15:41.949Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-12T03:15:42.210 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:41 smithi151 podman[64749]: 2024-03-12 03:15:41.958882127 +0000 UTC m=+0.130174333 container died 421b13c53e55401f2174f7f6b40f9e72adb55fe6d131f1eb88b8077a124f9b0f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:42.464 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:42 smithi151 podman[64749]: 2024-03-12 03:15:42.306736027 +0000 UTC m=+0.478028233 container cleanup 421b13c53e55401f2174f7f6b40f9e72adb55fe6d131f1eb88b8077a124f9b0f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:42.464 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:42 smithi151 bash[64749]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a 2024-03-12T03:15:42.940 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:42 smithi151 podman[64764]: 2024-03-12 03:15:42.575218117 +0000 UTC m=+0.619253030 container remove 421b13c53e55401f2174f7f6b40f9e72adb55fe6d131f1eb88b8077a124f9b0f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:42.940 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:42 smithi151 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-12T03:15:42.940 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:42 smithi151 systemd[1]: Stopped Ceph prometheus.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:15:42.940 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:42 smithi151 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@prometheus.a.service: Consumed 1.128s CPU time. 2024-03-12T03:15:42.940 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:42 smithi151 systemd[1]: Starting Ceph prometheus.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:15:43.207 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:42 smithi151 podman[64859]: 2024-03-12 03:15:42.842004049 +0000 UTC m=+0.019312742 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-12T03:15:43.208 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:42 smithi151 podman[64859]: 2024-03-12 03:15:42.959974326 +0000 UTC m=+0.137283014 container create af733172183c6dcbacc62caedbf96b30a6ce278a8f0354265f18d4c2c5e2fe6b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:43.208 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 podman[64859]: 2024-03-12 03:15:43.179648791 +0000 UTC m=+0.356957478 container init af733172183c6dcbacc62caedbf96b30a6ce278a8f0354265f18d4c2c5e2fe6b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:43.208 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 podman[64859]: 2024-03-12 03:15:43.182264883 +0000 UTC m=+0.359573574 container start af733172183c6dcbacc62caedbf96b30a6ce278a8f0354265f18d4c2c5e2fe6b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:15:43.465 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.205Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-12T03:15:43.465 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.206Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-12T03:15:43.465 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.206Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi151 (none))" 2024-03-12T03:15:43.465 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.206Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-12T03:15:43.465 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.206Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-12T03:15:43.465 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.207Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-12T03:15:43.466 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.207Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-12T03:15:43.466 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.208Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-12T03:15:43.466 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.208Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-12T03:15:43.466 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.211Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-12T03:15:43.466 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.211Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.912µs 2024-03-12T03:15:43.466 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.211Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-12T03:15:43.466 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.211Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-12T03:15:43.466 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.211Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-12T03:15:43.466 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.211Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.502µs wal_replay_duration=462.384µs wbl_replay_duration=115ns total_replay_duration=499.795µs 2024-03-12T03:15:43.466 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.212Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-12T03:15:43.467 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.212Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-12T03:15:43.467 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.212Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-12T03:15:43.467 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.232Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.162592ms db_storage=1.084µs remote_storage=1.02µs web_handler=345ns query_engine=590ns scrape=379.335µs scrape_sd=96.291µs notify=21.81µs notify_sd=23.592µs rules=19.35071ms tracing=6.376µs 2024-03-12T03:15:43.467 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.232Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-12T03:15:43.467 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:15:43.232Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-12T03:15:43.467 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 bash[64859]: af733172183c6dcbacc62caedbf96b30a6ce278a8f0354265f18d4c2c5e2fe6b 2024-03-12T03:15:43.467 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:15:43 smithi151 systemd[1]: Started Ceph prometheus.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:15:43.701 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Bus STOPPING 2024-03-12T03:15:43.702 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:15:43.702 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Bus STOPPED 2024-03-12T03:15:43.702 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Bus STARTING 2024-03-12T03:15:43.702 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Serving on http://:::9283 2024-03-12T03:15:43.702 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Bus STARTED 2024-03-12T03:15:43.702 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Bus STOPPING 2024-03-12T03:15:43.702 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:15:43.702 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Bus STOPPED 2024-03-12T03:15:43.702 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Bus STARTING 2024-03-12T03:15:43.702 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Serving on http://:::9283 2024-03-12T03:15:43.702 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Bus STARTED 2024-03-12T03:15:43.702 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Bus STOPPING 2024-03-12T03:15:43.702 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:15:43.703 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Bus STOPPED 2024-03-12T03:15:43.703 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Bus STARTING 2024-03-12T03:15:43.703 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Serving on http://:::9283 2024-03-12T03:15:43.703 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:43 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:15:43] ENGINE Bus STARTED 2024-03-12T03:15:44.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:44.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:15:44.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:15:44.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi094.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:15:44.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi094.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:15:44.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:15:44.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:15:44.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi151.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:15:44.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi151.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:15:44.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:15:44.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:15:44.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi151.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:15:44.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi151.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:15:44.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:44 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:15:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:15:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:15:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi094.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:15:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi094.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:15:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:15:44.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:15:44.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi151.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:15:44.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi151.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:15:44.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:15:44.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:15:44.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi151.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:15:44.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi151.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:15:44.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:15:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi094.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi094.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi151.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi151.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi151.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:15:44.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.94:0/603352780' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi151.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:15:44.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:44.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:44 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:15:46.623 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:46 smithi151 ceph-mon[32923]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:46.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:46 smithi094 ceph-mon[29291]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:46 smithi094 ceph-mon[40484]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:47 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:15:47.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:47 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:15:47.620 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:47 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:15:47.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:15:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:15:48.709 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:15:48 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[77353]: ts=2024-03-12T03:15:48.232Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004339186s 2024-03-12T03:15:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:48 smithi094 ceph-mon[29291]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:48 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:48 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:48 smithi094 ceph-mon[40484]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:48 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:48 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:48 smithi151 ceph-mon[32923]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:48 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:48 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:49.634 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:49.635 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:50.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:50 smithi094 ceph-mon[29291]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:50 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:50 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:15:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:15:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:50 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:50 smithi094 ceph-mon[40484]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:50 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:50 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:15:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:15:50.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:50 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:50.711 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:15:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:15:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:15:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:50 smithi151 ceph-mon[32923]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:50 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:50 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:15:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:15:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:50 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:15:52.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:52 smithi094 ceph-mon[40484]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:52 smithi094 ceph-mon[29291]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:52 smithi151 ceph-mon[32923]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:54.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:54 smithi094 ceph-mon[40484]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:54 smithi094 ceph-mon[29291]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:54 smithi151 ceph-mon[32923]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:15:56.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:56 smithi094 ceph-mon[29291]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:56 smithi094 ceph-mon[40484]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:56 smithi151 ceph-mon[32923]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:57.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:15:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:15:58.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:58 smithi094 ceph-mon[29291]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:58 smithi094 ceph-mon[40484]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:58 smithi151 ceph-mon[32923]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:15:59.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:15:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:15:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:15:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:15:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:00.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:00 smithi094 ceph-mon[29291]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:00 smithi094 ceph-mon[40484]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:00.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:16:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:16:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:16:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:00 smithi151 ceph-mon[32923]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:02.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:02 smithi094 ceph-mon[40484]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:02 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:16:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:02 smithi094 ceph-mon[29291]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:02 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:16:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:02 smithi151 ceph-mon[32923]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:02 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:16:04.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:04 smithi094 ceph-mon[40484]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:04 smithi094 ceph-mon[29291]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:04 smithi151 ceph-mon[32923]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:06.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:06 smithi094 ceph-mon[40484]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:06 smithi094 ceph-mon[29291]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:06 smithi151 ceph-mon[32923]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:07.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:16:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:16:08.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:08 smithi094 ceph-mon[40484]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:08.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:08 smithi094 ceph-mon[29291]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:08 smithi151 ceph-mon[32923]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:09.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:10.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:10 smithi094 ceph-mon[40484]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:10.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:16:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:16:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:16:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:10 smithi094 ceph-mon[29291]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:10 smithi151 ceph-mon[32923]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:12.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:12 smithi094 ceph-mon[40484]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:12 smithi094 ceph-mon[29291]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:12 smithi151 ceph-mon[32923]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:14.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:14 smithi094 ceph-mon[29291]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:14 smithi094 ceph-mon[40484]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:14 smithi151 ceph-mon[32923]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:15.550 INFO:tasks.workunit.client.0.smithi094.stderr:Note: switching to '1c349451176cc5b4ebfb24b22eaaa754e05cff6c'. 2024-03-12T03:16:15.550 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-03-12T03:16:15.550 INFO:tasks.workunit.client.0.smithi094.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr:state without impacting any branches by switching back to a branch. 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr: git switch -c 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr:Or undo this operation with: 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr: git switch - 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-03-12T03:16:15.551 INFO:tasks.workunit.client.0.smithi094.stderr:HEAD is now at 1c349451176 Merge pull request #56077 from idryomov/wip-rbd-mirror-bash 2024-03-12T03:16:15.553 DEBUG:teuthology.orchestra.run.smithi094:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-12T03:16:15.596 INFO:tasks.workunit.client.0.smithi094.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-12T03:16:15.598 INFO:tasks.workunit.client.0.smithi094.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-12T03:16:15.598 INFO:tasks.workunit.client.0.smithi094.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-12T03:16:16.372 INFO:tasks.workunit.client.0.smithi094.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-12T03:16:16.435 INFO:tasks.workunit.client.0.smithi094.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-12T03:16:16.475 INFO:tasks.workunit.client.0.smithi094.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-12T03:16:16.477 INFO:tasks.workunit.client.0.smithi094.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-12T03:16:16.477 INFO:tasks.workunit.client.0.smithi094.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-12T03:16:16.515 INFO:tasks.workunit.client.0.smithi094.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-12T03:16:16.518 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-12T03:16:16.519 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-12T03:16:16.574 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-12T03:16:16.575 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-12T03:16:16.575 DEBUG:teuthology.orchestra.run.smithi094:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-12T03:16:16.636 INFO:tasks.workunit.client.0.smithi094.stderr:+ parallel=1 2024-03-12T03:16:16.637 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' '' = --serial ']' 2024-03-12T03:16:16.638 INFO:tasks.workunit.client.0.smithi094.stderr:+ crimson=0 2024-03-12T03:16:16.638 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' '' = --crimson ']' 2024-03-12T03:16:16.638 INFO:tasks.workunit.client.0.smithi094.stderr:+ color= 2024-03-12T03:16:16.638 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -t 1 ']' 2024-03-12T03:16:16.638 INFO:tasks.workunit.client.0.smithi094.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-12T03:16:16.638 INFO:tasks.workunit.client.0.smithi094.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-12T03:16:16.638 INFO:tasks.workunit.client.0.smithi094.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-12T03:16:16.639 INFO:tasks.workunit.client.0.smithi094.stderr:+ declare -A pids 2024-03-12T03:16:16.640 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.640 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.640 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_aio 2024-03-12T03:16:16.640 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_aio' 2024-03-12T03:16:16.640 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_aio 2024-03-12T03:16:16.640 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.640 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_aio 2024-03-12T03:16:16.640 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78013 2024-03-12T03:16:16.640 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_aio on pid 78013' 2024-03-12T03:16:16.640 INFO:tasks.workunit.client.0.smithi094.stdout:test api_aio on pid 78013 2024-03-12T03:16:16.643 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78013 2024-03-12T03:16:16.643 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.643 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.643 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-12T03:16:16.643 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_aio_pp 2024-03-12T03:16:16.643 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_aio_pp' 2024-03-12T03:16:16.643 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_aio_pp 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.644 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.645 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.645 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.645 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.645 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.645 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.645 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.645 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.645 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.645 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.645 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.645 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.645 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_aio_pp 2024-03-12T03:16:16.645 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78020 2024-03-12T03:16:16.646 INFO:tasks.workunit.client.0.smithi094.stdout:test api_aio_pp on pid 78020 2024-03-12T03:16:16.649 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_aio_pp on pid 78020' 2024-03-12T03:16:16.649 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78020 2024-03-12T03:16:16.649 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.649 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_io 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_io' 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_io 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.652 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.653 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.653 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.653 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.653 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.653 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.653 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.653 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_io 2024-03-12T03:16:16.653 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78029 2024-03-12T03:16:16.653 INFO:tasks.workunit.client.0.smithi094.stdout:test api_io on pid 78029 2024-03-12T03:16:16.657 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_io on pid 78029' 2024-03-12T03:16:16.657 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78029 2024-03-12T03:16:16.657 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.657 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.657 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-12T03:16:16.657 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_io_pp 2024-03-12T03:16:16.657 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_io_pp' 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_io_pp 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.658 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.659 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.660 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.661 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.661 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.661 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.661 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.661 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.661 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.661 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.661 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.661 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.661 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.661 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_io_pp 2024-03-12T03:16:16.661 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78040 2024-03-12T03:16:16.662 INFO:tasks.workunit.client.0.smithi094.stdout:test api_io_pp on pid 78040 2024-03-12T03:16:16.668 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_io_pp on pid 78040' 2024-03-12T03:16:16.668 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78040 2024-03-12T03:16:16.668 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.668 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.668 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-12T03:16:16.668 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_asio 2024-03-12T03:16:16.668 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_asio' 2024-03-12T03:16:16.668 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_asio 2024-03-12T03:16:16.668 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.668 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.668 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.669 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.670 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.671 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.672 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_asio 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.674 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.674 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78054 2024-03-12T03:16:16.674 INFO:tasks.workunit.client.0.smithi094.stdout:test api_asio on pid 78054 2024-03-12T03:16:16.680 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_asio on pid 78054' 2024-03-12T03:16:16.680 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78054 2024-03-12T03:16:16.680 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-12T03:16:16.680 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.680 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.680 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.680 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_list 2024-03-12T03:16:16.680 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_list' 2024-03-12T03:16:16.680 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_list 2024-03-12T03:16:16.680 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.680 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.680 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.680 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.681 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.683 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.685 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.686 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.687 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.687 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.687 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.687 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.687 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78013/exe 2024-03-12T03:16:16.687 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_list 2024-03-12T03:16:16.687 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78071 2024-03-12T03:16:16.687 INFO:tasks.workunit.client.0.smithi094.stdout:test api_list on pid 78071 2024-03-12T03:16:16.692 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_list on pid 78071' 2024-03-12T03:16:16.692 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78071 2024-03-12T03:16:16.692 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.692 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.692 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-12T03:16:16.692 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_lock 2024-03-12T03:16:16.692 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_lock' 2024-03-12T03:16:16.692 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.692 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.692 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.692 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_lock 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.694 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_aio: /' 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.696 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.697 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.698 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.700 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.700 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.700 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.700 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.700 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.700 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.700 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.700 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.700 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78020/exe 2024-03-12T03:16:16.700 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.700 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.700 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.700 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_lock 2024-03-12T03:16:16.701 INFO:tasks.workunit.client.0.smithi094.stdout:test api_lock on pid 78093 2024-03-12T03:16:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78093 2024-03-12T03:16:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_lock on pid 78093' 2024-03-12T03:16:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78093 2024-03-12T03:16:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-12T03:16:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_lock_pp 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_lock_pp' 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_lock_pp 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78029/exe 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.712 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.712 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_lock_pp 2024-03-12T03:16:16.712 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-12T03:16:16.712 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.712 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78115 2024-03-12T03:16:16.712 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_lock_pp on pid 78115' 2024-03-12T03:16:16.712 INFO:tasks.workunit.client.0.smithi094.stdout:test api_lock_pp on pid 78115 2024-03-12T03:16:16.713 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:16 smithi151 ceph-mon[32923]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:16.716 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.716 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78115 2024-03-12T03:16:16.716 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.716 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.716 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.716 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.716 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_io: /' 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_io.log 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_misc 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_misc' 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78040/exe 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_misc 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-12T03:16:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-12T03:16:16.727 INFO:tasks.workunit.client.0.smithi094.stdout:test api_misc on pid 78141 2024-03-12T03:16:16.728 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.728 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.728 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.728 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_misc 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78141 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_misc on pid 78141' 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78141 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_misc_pp 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_misc_pp' 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-12T03:16:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78054/exe 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_misc_pp 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_misc_pp 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78159 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_misc_pp on pid 78159' 2024-03-12T03:16:16.736 INFO:tasks.workunit.client.0.smithi094.stdout:test api_misc_pp on pid 78159 2024-03-12T03:16:16.738 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78159 2024-03-12T03:16:16.738 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.738 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.738 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.738 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-12T03:16:16.738 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_tier_pp 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_tier_pp' 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_asio: /' 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78071/exe 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_tier_pp 2024-03-12T03:16:16.744 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_tier_pp 2024-03-12T03:16:16.744 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78093/exe 2024-03-12T03:16:16.744 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.744 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78174 2024-03-12T03:16:16.744 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_tier_pp on pid 78174' 2024-03-12T03:16:16.744 INFO:tasks.workunit.client.0.smithi094.stdout:test api_tier_pp on pid 78174 2024-03-12T03:16:16.747 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78174 2024-03-12T03:16:16.747 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.747 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.747 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.747 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.747 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.747 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_list: /' 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_pool 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_pool' 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-03-12T03:16:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-03-12T03:16:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-12T03:16:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_list.log 2024-03-12T03:16:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_lock: /' 2024-03-12T03:16:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.757 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.757 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.758 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.759 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.760 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.761 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.761 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.761 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_pool 2024-03-12T03:16:16.761 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.761 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.761 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.761 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.761 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.761 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.761 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_pool 2024-03-12T03:16:16.761 INFO:tasks.workunit.client.0.smithi094.stdout:test api_pool on pid 78227 2024-03-12T03:16:16.764 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78227 2024-03-12T03:16:16.764 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_pool on pid 78227' 2024-03-12T03:16:16.764 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78227 2024-03-12T03:16:16.764 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.764 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.764 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_snapshots 2024-03-12T03:16:16.764 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-12T03:16:16.764 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_snapshots' 2024-03-12T03:16:16.764 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.764 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.764 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_snapshots 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.766 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.766 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.766 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.766 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.766 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.766 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.766 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.766 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.766 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.766 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.766 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.766 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.767 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.767 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:16 smithi094 ceph-mon[29291]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:16.767 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:16 smithi094 ceph-mon[40484]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:16.770 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78115/exe 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.771 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-03-12T03:16:16.772 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.773 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.773 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.773 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.773 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.773 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.773 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.773 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-12T03:16:16.773 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.773 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-12T03:16:16.773 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.773 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.773 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.773 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.774 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.775 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.775 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.775 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.775 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.775 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.775 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.775 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78141/exe 2024-03-12T03:16:16.775 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78159/exe 2024-03-12T03:16:16.775 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.775 INFO:tasks.workunit.client.0.smithi094.stdout:test api_snapshots on pid 78279 2024-03-12T03:16:16.778 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.778 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.778 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.778 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.778 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.778 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_snapshots 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78279 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_snapshots on pid 78279' 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78279 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_snapshots_pp 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_snapshots_pp' 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_misc: /' 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.782 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.782 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.782 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.782 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.782 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.782 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.782 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.782 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.782 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.782 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.782 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.782 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.782 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_snapshots_pp 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.784 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.784 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.784 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.784 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.784 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.784 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.784 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.784 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.784 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.784 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.784 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.785 INFO:tasks.workunit.client.0.smithi094.stdout:test api_snapshots_pp on pid 78300 2024-03-12T03:16:16.788 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_snapshots_pp 2024-03-12T03:16:16.788 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78300 2024-03-12T03:16:16.788 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_snapshots_pp on pid 78300' 2024-03-12T03:16:16.788 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78300 2024-03-12T03:16:16.788 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.789 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.790 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_stat 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_stat' 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.791 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.792 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.792 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.792 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.792 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.792 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.792 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.792 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.792 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.792 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.792 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.792 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.792 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.793 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.793 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.793 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.793 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-03-12T03:16:16.793 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.793 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.793 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.793 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.793 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.793 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.793 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-12T03:16:16.793 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.793 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_stat 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_stat 2024-03-12T03:16:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78344 2024-03-12T03:16:16.797 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_stat on pid 78344' 2024-03-12T03:16:16.797 INFO:tasks.workunit.client.0.smithi094.stdout:test api_stat on pid 78344 2024-03-12T03:16:16.803 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78344 2024-03-12T03:16:16.803 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.803 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.803 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.803 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.803 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.803 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.803 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.805 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.806 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.807 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_stat_pp 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_stat_pp' 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78227/exe 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.808 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_stat_pp 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.809 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.812 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_pool: /' 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.813 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.815 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.816 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_stat_pp 2024-03-12T03:16:16.816 INFO:tasks.workunit.client.0.smithi094.stdout:test api_stat_pp on pid 78386 2024-03-12T03:16:16.820 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78386 2024-03-12T03:16:16.820 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_stat_pp on pid 78386' 2024-03-12T03:16:16.820 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78386 2024-03-12T03:16:16.820 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.820 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.820 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-12T03:16:16.820 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.824 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.825 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.826 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.827 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_watch_notify 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_watch_notify' 2024-03-12T03:16:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78279/exe 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_watch_notify 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.831 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.832 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.833 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78300/exe 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.834 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.835 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-03-12T03:16:16.835 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_watch_notify 2024-03-12T03:16:16.835 INFO:tasks.workunit.client.0.smithi094.stdout:test api_watch_notify on pid 78443 2024-03-12T03:16:16.838 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78443 2024-03-12T03:16:16.839 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_watch_notify on pid 78443' 2024-03-12T03:16:16.839 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78443 2024-03-12T03:16:16.839 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-12T03:16:16.839 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.839 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.839 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-12T03:16:16.839 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-12T03:16:16.839 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.839 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.839 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.839 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.839 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.839 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78174/exe 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.840 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.841 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.842 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.843 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.844 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.845 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_watch_notify_pp 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78344/exe 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_watch_notify_pp' 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.846 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.847 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.847 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.847 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.847 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.847 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.847 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.847 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.847 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.847 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.847 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.847 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.847 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.848 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.848 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.848 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.848 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.848 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_watch_notify_pp 2024-03-12T03:16:16.848 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.848 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.848 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.848 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.848 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.848 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.848 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.848 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.849 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.849 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.849 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_stat: /' 2024-03-12T03:16:16.849 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-03-12T03:16:16.849 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.849 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-12T03:16:16.849 INFO:tasks.workunit.client.0.smithi094.stdout:test api_watch_notify_pp on pid 78476 2024-03-12T03:16:16.851 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_watch_notify_pp 2024-03-12T03:16:16.851 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78476 2024-03-12T03:16:16.851 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_watch_notify_pp on pid 78476' 2024-03-12T03:16:16.851 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78476 2024-03-12T03:16:16.851 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.851 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.851 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-12T03:16:16.851 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.851 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.851 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.851 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.852 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.852 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.852 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_cmd 2024-03-12T03:16:16.852 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_cmd' 2024-03-12T03:16:16.852 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_cmd 2024-03-12T03:16:16.852 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.852 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.852 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.852 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.852 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.852 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.852 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.853 INFO:tasks.workunit.client.0.smithi094.stdout:test api_cmd on pid 78495 2024-03-12T03:16:16.855 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_cmd 2024-03-12T03:16:16.855 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.855 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.855 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78495 2024-03-12T03:16:16.855 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.855 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_cmd on pid 78495' 2024-03-12T03:16:16.855 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78495 2024-03-12T03:16:16.855 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.855 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.855 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.856 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_cmd_pp 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_cmd_pp' 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_cmd_pp 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.863 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.864 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.865 INFO:tasks.workunit.client.0.smithi094.stdout:test api_cmd_pp on pid 78554 2024-03-12T03:16:16.867 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_cmd_pp 2024-03-12T03:16:16.867 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78554 2024-03-12T03:16:16.867 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_cmd_pp on pid 78554' 2024-03-12T03:16:16.867 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78554 2024-03-12T03:16:16.867 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_service 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_service' 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_service 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.869 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.870 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.870 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.870 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.870 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.870 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.870 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.870 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.870 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.870 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.870 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.870 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.870 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.870 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78386/exe 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.871 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.872 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.872 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.872 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.872 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.872 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.872 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.872 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.872 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.872 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.872 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.872 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.872 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.872 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.873 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.873 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.873 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.873 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.873 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.873 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.873 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.873 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.873 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.873 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.873 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.873 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-12T03:16:16.874 INFO:tasks.workunit.client.0.smithi094.stdout:test api_service on pid 78586 2024-03-12T03:16:16.875 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_service 2024-03-12T03:16:16.875 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78586 2024-03-12T03:16:16.875 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_service on pid 78586' 2024-03-12T03:16:16.876 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78586 2024-03-12T03:16:16.876 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.876 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.876 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78443/exe 2024-03-12T03:16:16.876 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-03-12T03:16:16.876 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_service_pp 2024-03-12T03:16:16.876 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-12T03:16:16.876 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_service_pp' 2024-03-12T03:16:16.876 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-12T03:16:16.876 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.876 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.876 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.876 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_service_pp 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.877 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_service_pp 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.879 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.880 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.880 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.880 INFO:tasks.workunit.client.0.smithi094.stdout:test api_service_pp on pid 78599 2024-03-12T03:16:16.884 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.884 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.884 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78599 2024-03-12T03:16:16.884 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_service_pp on pid 78599' 2024-03-12T03:16:16.884 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78599 2024-03-12T03:16:16.884 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.884 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.884 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.884 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.884 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.884 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.884 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_c_write_operations 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_c_write_operations' 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.885 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.886 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_c_write_operations 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.887 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_c_write_operations 2024-03-12T03:16:16.891 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-12T03:16:16.892 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78619 2024-03-12T03:16:16.892 INFO:tasks.workunit.client.0.smithi094.stdout:test api_c_write_operations on pid 78619 2024-03-12T03:16:16.895 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_c_write_operations on pid 78619' 2024-03-12T03:16:16.895 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78619 2024-03-12T03:16:16.895 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78476/exe 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_c_read_operations 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-12T03:16:16.896 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_c_read_operations' 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.897 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.898 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-12T03:16:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-12T03:16:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_c_read_operations 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.902 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.903 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.903 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_c_read_operations 2024-03-12T03:16:16.903 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.903 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.903 INFO:tasks.workunit.client.0.smithi094.stdout:test api_c_read_operations on pid 78655 2024-03-12T03:16:16.905 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.906 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.906 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.906 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.906 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.906 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78655 2024-03-12T03:16:16.906 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_c_read_operations on pid 78655' 2024-03-12T03:16:16.906 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78655 2024-03-12T03:16:16.906 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.906 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.906 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.906 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.906 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.907 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.907 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.907 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.907 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.907 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.907 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.907 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.907 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.907 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.907 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.907 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s api_cls_remote_reads 2024-03-12T03:16:16.907 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-12T03:16:16.907 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' api_cls_remote_reads' 2024-03-12T03:16:16.908 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.908 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78495/exe 2024-03-12T03:16:16.908 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.908 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.908 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.908 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.908 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.908 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.908 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo api_cls_remote_reads 2024-03-12T03:16:16.908 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.908 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.908 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.909 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.910 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.911 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=api_cls_remote_reads 2024-03-12T03:16:16.912 INFO:tasks.workunit.client.0.smithi094.stdout:test api_cls_remote_reads on pid 78686 2024-03-12T03:16:16.914 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78686 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test api_cls_remote_reads on pid 78686' 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78686 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.915 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.916 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s list_parallel 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' list_parallel' 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.917 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.918 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.921 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo list_parallel 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_cmd: /' 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.922 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.923 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.924 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.924 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.924 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.924 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.924 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.924 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.924 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.924 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.924 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.924 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.924 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.924 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.924 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78554/exe 2024-03-12T03:16:16.925 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.925 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.925 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.925 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.925 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.925 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=list_parallel 2024-03-12T03:16:16.925 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78709 2024-03-12T03:16:16.925 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test list_parallel on pid 78709' 2024-03-12T03:16:16.925 INFO:tasks.workunit.client.0.smithi094.stdout:test list_parallel on pid 78709 2024-03-12T03:16:16.929 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78709 2024-03-12T03:16:16.929 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.929 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.929 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-12T03:16:16.929 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.929 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.929 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.929 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.929 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.929 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.929 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.930 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.931 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.932 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.933 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s open_pools_parallel 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' open_pools_parallel' 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.936 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.937 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.938 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo open_pools_parallel 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.939 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.940 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.941 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.941 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.941 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.941 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=open_pools_parallel 2024-03-12T03:16:16.941 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78740 2024-03-12T03:16:16.941 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test open_pools_parallel on pid 78740' 2024-03-12T03:16:16.941 INFO:tasks.workunit.client.0.smithi094.stdout:test open_pools_parallel on pid 78740 2024-03-12T03:16:16.947 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78740 2024-03-12T03:16:16.947 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-12T03:16:16.947 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.947 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s delete_pools_parallel 2024-03-12T03:16:16.947 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.947 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' delete_pools_parallel' 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.948 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.949 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.950 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.951 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.952 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.952 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.952 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.952 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.952 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.952 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.952 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.952 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.952 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.952 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.952 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.952 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78586/exe 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.953 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.954 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.955 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78599/exe 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78619/exe 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.956 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.957 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.957 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.957 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.957 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.957 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.957 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.957 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.957 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.957 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.957 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.957 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.957 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.957 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.958 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.958 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.958 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.958 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.958 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.958 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.958 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.958 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.958 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.958 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.958 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.958 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.958 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo delete_pools_parallel 2024-03-12T03:16:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:16.960 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.961 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.962 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.962 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.962 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.962 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.962 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.962 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.962 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.962 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.962 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.962 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_service: /' 2024-03-12T03:16:16.962 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.962 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-12T03:16:16.962 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-03-12T03:16:16.965 INFO:tasks.workunit.client.0.smithi094.stdout:test delete_pools_parallel on pid 78784 2024-03-12T03:16:16.973 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=delete_pools_parallel 2024-03-12T03:16:16.973 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78784 2024-03-12T03:16:16.973 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test delete_pools_parallel on pid 78784' 2024-03-12T03:16:16.973 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78784 2024-03-12T03:16:16.973 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-12T03:16:16.973 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.974 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s cls 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' cls' 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_service.log 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:16.975 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo cls 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78686/exe 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.976 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78655/exe 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.979 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.980 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.981 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78709/exe 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:16.982 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.984 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.984 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.984 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.984 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.984 INFO:tasks.workunit.client.0.smithi094.stdout:test cls on pid 78838 2024-03-12T03:16:16.986 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=cls 2024-03-12T03:16:16.986 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78838 2024-03-12T03:16:16.986 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test cls on pid 78838' 2024-03-12T03:16:16.986 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78838 2024-03-12T03:16:16.986 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-12T03:16:16.986 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:16.986 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.987 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s cmd 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' cmd' 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:16.988 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ list_parallel: /' 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.989 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo cmd 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:16.990 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:16.991 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:16.991 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:16.991 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:16.991 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:16.991 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:16.991 INFO:tasks.workunit.client.0.smithi094.stdout:test cmd on pid 78871 2024-03-12T03:16:16.999 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=cmd 2024-03-12T03:16:16.999 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78871 2024-03-12T03:16:16.999 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test cmd on pid 78871' 2024-03-12T03:16:16.999 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78871 2024-03-12T03:16:16.999 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-12T03:16:16.999 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.000 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.001 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s handler_error 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' handler_error' 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.002 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.003 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:17.004 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.005 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:17.006 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:17.007 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:17.007 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.007 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:17.007 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.007 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:17.007 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.007 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.007 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.007 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.007 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.007 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.007 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.007 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.008 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78784/exe 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-12T03:16:17.009 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:17.010 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.011 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.012 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:17.013 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:17.013 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.013 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo handler_error 2024-03-12T03:16:17.013 INFO:tasks.workunit.client.0.smithi094.stdout:test handler_error on pid 78983 2024-03-12T03:16:17.015 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=handler_error 2024-03-12T03:16:17.015 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78983 2024-03-12T03:16:17.015 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test handler_error on pid 78983' 2024-03-12T03:16:17.015 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=78983 2024-03-12T03:16:17.015 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-12T03:16:17.015 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:17.015 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.015 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s io 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' io' 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.017 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.017 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.017 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.017 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78740/exe 2024-03-12T03:16:17.017 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo io 2024-03-12T03:16:17.017 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.017 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:17.017 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.017 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-03-12T03:16:17.017 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.017 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.017 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.017 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:17.018 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.019 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:17.020 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.020 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:17.020 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.020 INFO:tasks.workunit.client.0.smithi094.stdout:test io on pid 79023 2024-03-12T03:16:17.024 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=io 2024-03-12T03:16:17.024 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79023 2024-03-12T03:16:17.024 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test io on pid 79023' 2024-03-12T03:16:17.024 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=79023 2024-03-12T03:16:17.024 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-12T03:16:17.024 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:17.024 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.024 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.024 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.024 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.024 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.024 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-03-12T03:16:17.025 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s ec_io 2024-03-12T03:16:17.025 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' ec_io' 2024-03-12T03:16:17.025 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:17.025 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:17.025 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.025 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:17.025 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:17.025 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.025 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:17.025 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:17.025 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:17.025 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:17.025 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:17.026 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.027 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.028 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.028 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.028 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.028 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.028 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.028 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.028 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.028 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.028 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.028 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.028 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.028 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.028 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo ec_io 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.029 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-03-12T03:16:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-12T03:16:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-12T03:16:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:17.031 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.031 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:17.031 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:17.031 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:17.031 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.031 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:17.031 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.031 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:17.031 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.031 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78871/exe 2024-03-12T03:16:17.031 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.031 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.031 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.032 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:17.033 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:17.033 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:17.033 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.033 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:17.033 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.033 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:17.033 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:17.033 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.033 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_neorados_cmd 2024-03-12T03:16:17.033 INFO:tasks.workunit.client.0.smithi094.stdout:test ec_io on pid 79068 2024-03-12T03:16:17.036 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=ec_io 2024-03-12T03:16:17.036 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79068 2024-03-12T03:16:17.036 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test ec_io on pid 79068' 2024-03-12T03:16:17.036 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=79068 2024-03-12T03:16:17.036 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-12T03:16:17.036 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:17.036 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.037 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s list 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' list' 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:17.038 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-03-12T03:16:17.039 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_neorados_cmd.log 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ cmd: /' 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78838/exe 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo list 2024-03-12T03:16:17.040 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.041 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ cls: /' 2024-03-12T03:16:17.042 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:17.042 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=list 2024-03-12T03:16:17.042 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79096 2024-03-12T03:16:17.042 INFO:tasks.workunit.client.0.smithi094.stdout:test list on pid 79096 2024-03-12T03:16:17.044 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test list on pid 79096' 2024-03-12T03:16:17.044 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=79096 2024-03-12T03:16:17.044 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-12T03:16:17.044 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:17.044 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_neorados_cls.log 2024-03-12T03:16:17.044 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s ec_list 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.045 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' ec_list' 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_neorados_cls 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:17.046 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:17.047 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo ec_list 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.049 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.049 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.049 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.049 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.049 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.049 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=ec_list 2024-03-12T03:16:17.049 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79119 2024-03-12T03:16:17.049 INFO:tasks.workunit.client.0.smithi094.stdout:test ec_list on pid 79119 2024-03-12T03:16:17.056 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test ec_list on pid 79119' 2024-03-12T03:16:17.056 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=79119 2024-03-12T03:16:17.056 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-12T03:16:17.056 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:17.056 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.056 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.056 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.056 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.056 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:17.057 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:17.058 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:17.058 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.058 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:17.058 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.058 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:17.058 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:17.058 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:17.058 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.058 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:17.058 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.058 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:17.058 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:17.058 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:17.059 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s misc 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' misc' 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/78983/exe 2024-03-12T03:16:17.065 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.066 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo misc 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.068 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_neorados_handler_error 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_neorados_handler_error.log 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/79023/exe 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.070 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.070 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.070 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.070 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=misc 2024-03-12T03:16:17.070 INFO:tasks.workunit.client.0.smithi094.stdout:test misc on pid 79200 2024-03-12T03:16:17.073 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.074 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.074 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.074 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79200 2024-03-12T03:16:17.074 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test misc on pid 79200' 2024-03-12T03:16:17.074 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=79200 2024-03-12T03:16:17.074 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-12T03:16:17.074 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:17.074 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ io: /' 2024-03-12T03:16:17.074 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-03-12T03:16:17.074 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_neorados_io.log 2024-03-12T03:16:17.074 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_neorados_io 2024-03-12T03:16:17.074 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:17.074 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:17.075 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:17.075 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:17.075 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.075 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:17.075 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.075 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.075 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.075 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.075 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.075 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.075 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.075 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.075 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ handler_error: /' 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:17.076 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.077 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:17.078 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.078 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.078 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.078 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.078 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.078 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.078 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:17.078 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:17.078 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:17.078 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.078 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:17.078 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.078 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s pool 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.079 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' pool' 2024-03-12T03:16:17.080 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.080 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.080 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:17.080 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:17.080 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:17.080 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.080 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:17.080 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.080 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:17.080 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:17.080 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:17.080 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.080 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.081 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo pool 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:17.082 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:17.083 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.083 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:17.083 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.083 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.083 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:17.083 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.083 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:17.083 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.083 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.083 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.083 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.083 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.083 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/79096/exe 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.084 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.085 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.086 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=pool 2024-03-12T03:16:17.086 INFO:tasks.workunit.client.0.smithi094.stdout:test pool on pid 79251 2024-03-12T03:16:17.088 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79251 2024-03-12T03:16:17.088 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test pool on pid 79251' 2024-03-12T03:16:17.088 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=79251 2024-03-12T03:16:17.088 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-12T03:16:17.088 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:17.088 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:17.089 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s read_operations 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' read_operations' 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ list: /' 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/79068/exe 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.090 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.091 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.092 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_neorados_list.log 2024-03-12T03:16:17.092 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.092 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.092 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.092 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.092 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.092 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.092 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.092 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.092 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.092 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo read_operations 2024-03-12T03:16:17.092 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-03-12T03:16:17.092 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_neorados_list 2024-03-12T03:16:17.093 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.093 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.093 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.093 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.093 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.093 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.093 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.093 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_neorados_ec_io.log 2024-03-12T03:16:17.093 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ ec_io: /' 2024-03-12T03:16:17.093 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=read_operations 2024-03-12T03:16:17.093 INFO:tasks.workunit.client.0.smithi094.stdout:test read_operations on pid 79285 2024-03-12T03:16:17.097 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79285 2024-03-12T03:16:17.097 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test read_operations on pid 79285' 2024-03-12T03:16:17.097 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=79285 2024-03-12T03:16:17.097 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-12T03:16:17.097 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:17.097 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.097 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.097 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:17.097 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.097 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:17.097 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.098 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.098 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.098 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:17.098 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.098 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:17.098 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.098 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.098 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.098 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:17.098 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.098 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.098 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_neorados_ec_io 2024-03-12T03:16:17.098 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:17.099 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.101 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.102 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.102 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.102 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.102 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.102 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.102 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.102 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.102 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.102 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.102 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.102 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s snapshots 2024-03-12T03:16:17.102 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' snapshots' 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo snapshots 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/79119/exe 2024-03-12T03:16:17.105 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=snapshots 2024-03-12T03:16:17.105 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:17.105 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:17.105 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:17.105 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.105 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79327 2024-03-12T03:16:17.105 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:17.105 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test snapshots on pid 79327' 2024-03-12T03:16:17.105 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.105 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:17.105 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=79327 2024-03-12T03:16:17.105 INFO:tasks.workunit.client.0.smithi094.stdout:test snapshots on pid 79327 2024-03-12T03:16:17.107 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-12T03:16:17.107 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:17.107 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:17.107 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s watch_notify 2024-03-12T03:16:17.108 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' watch_notify' 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/79200/exe 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo watch_notify 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-03-12T03:16:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.111 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.112 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.112 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.112 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_neorados_misc 2024-03-12T03:16:17.112 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_neorados_ec_list 2024-03-12T03:16:17.112 INFO:tasks.workunit.client.0.smithi094.stdout:test watch_notify on pid 79357 2024-03-12T03:16:17.116 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=watch_notify 2024-03-12T03:16:17.116 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79357 2024-03-12T03:16:17.116 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test watch_notify on pid 79357' 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=79357 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_neorados_ec_list.log 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ ec_list: /' 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ misc: /' 2024-03-12T03:16:17.117 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:17.118 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:17.118 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.118 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:17.118 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.118 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:17.118 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:17.118 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.118 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:17.118 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:17.118 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:17.118 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.118 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:17.119 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:17.120 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_neorados_misc.log 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:++ printf %25s write_operations 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:+ r=' write_operations' 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:++ awk '{print $1}' 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:++ echo write_operations 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.121 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:17.122 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.123 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.123 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.123 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:17.123 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.123 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:17.123 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.123 INFO:tasks.workunit.client.0.smithi094.stdout:test write_operations on pid 79381 2024-03-12T03:16:17.136 INFO:tasks.workunit.client.0.smithi094.stderr:+ ff=write_operations 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79381 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo 'test write_operations on pid 79381' 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:+ pids[$f]=79381 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:+ ret=0 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 1 -eq 1 ']' 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78174 2024-03-12T03:16:17.137 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78174 2024-03-12T03:16:17.138 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:17.138 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.138 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.138 INFO:tasks.workunit.client.0.smithi094.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-03-12T03:16:17.138 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.138 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.138 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.138 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.138 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.138 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.138 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.138 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.138 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -f /etc/bashrc ']' 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:+ . /etc/bashrc 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.140 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -z '' ']' 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:++ BASHRCSOURCED=Y 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:++ shopt -q login_shell 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:17.141 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.142 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:17.142 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:17.142 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:17.142 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.142 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:17.142 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.142 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:17.142 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.142 INFO:tasks.workunit.client.0.smithi094.stderr:+++ umask 2024-03-12T03:16:17.142 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' 0022 -eq 0 ']' 2024-03-12T03:16:17.142 INFO:tasks.workunit.client.0.smithi094.stderr:++ SHELL=/bin/bash 2024-03-12T03:16:17.142 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-12T03:16:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'grep=grep --color=auto' 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.147 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorls.sh 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' '!' -t 0 ']' 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:+++ return 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.148 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/79251/exe 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export S_COLORS=auto 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:+++ S_COLORS=auto 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-12T03:16:17.149 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.150 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.151 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.152 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.153 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.153 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.153 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.153 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.153 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.153 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.153 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.153 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.153 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.153 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.153 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.153 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.153 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/79285/exe 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_neorados_pool 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_neorados_pool.log 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ pool: /' 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-12T03:16:17.154 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.155 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.156 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_neorados_read_operations 2024-03-12T03:16:17.157 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.158 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.158 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.158 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.158 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.158 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ read_operations: /' 2024-03-12T03:16:17.158 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_neorados_read_operations.log 2024-03-12T03:16:17.158 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/79327/exe 2024-03-12T03:16:17.158 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.158 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.158 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.158 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.158 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.158 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/gawk.sh 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/lang.sh 2024-03-12T03:16:17.159 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/locale 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.160 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.161 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/79357/exe 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_neorados_snapshots.log 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ snapshots: /' 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.162 INFO:tasks.workunit.client.0.smithi094.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:++++ export LANG=en_US.UTF-8 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:++++ LANG=en_US.UTF-8 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:+++ LANG=en_US.UTF-8 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:+++ unset LANG_backup config 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n '' ']' 2024-03-12T03:16:17.163 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' dumb = linux ']' 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/less.sh 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:++ for i in /etc/profile.d/*.sh 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:++ '[' '' ']' 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:++ . /etc/profile.d/which2.sh 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_neorados_watch_notify 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_neorados_snapshots 2024-03-12T03:16:17.164 INFO:tasks.workunit.client.0.smithi094.stderr:+++++ readlink /proc/79381/exe 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:++++ basename /usr/bin/bash 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ watch_notify: /' 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_declare='declare -f' 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:+++ which_opt=-f 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export which_declare 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:+++ export -f which 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset i 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:++ unset -f pathmunge 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:+ export PATH 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:+ unset rc 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph_test_neorados_write_operations 2024-03-12T03:16:17.165 INFO:tasks.workunit.client.0.smithi094.stderr:+ tee ceph_test_neorados_write_operations.log 2024-03-12T03:16:17.166 INFO:tasks.workunit.client.0.smithi094.stderr:+ sed 's/^/ write_operations: /' 2024-03-12T03:16:17.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:16:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/67093169' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-12T03:16:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1693340182' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi094-79269-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2469088620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi094-79350-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:16:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/67093169' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-12T03:16:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1693340182' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi094-79269-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2469088620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi094-79350-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:17.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:16:17 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:16:17.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:16:17.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/67093169' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-12T03:16:17.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:17.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1693340182' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi094-79269-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:17.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2469088620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi094-79350-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:20.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:16:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:16:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:16:24.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[29291]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/389954204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi094-78138-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2366123043' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi094-78118-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/198683548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi094-78081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/43213594' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-12T03:16:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[29291]: osdmap e73: 8 total, 8 up, 8 in 2024-03-12T03:16:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[29291]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[29291]: from='client.34673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi094-78138-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[40484]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/389954204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi094-78138-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2366123043' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi094-78118-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/198683548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi094-78081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/43213594' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-12T03:16:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[40484]: osdmap e73: 8 total, 8 up, 8 in 2024-03-12T03:16:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[40484]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:23 smithi094 ceph-mon[40484]: from='client.34673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi094-78138-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:23 smithi151 ceph-mon[32923]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:16:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/389954204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi094-78138-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2366123043' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi094-78118-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/198683548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi094-78081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:23 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:24.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/43213594' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-12T03:16:24.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:23 smithi151 ceph-mon[32923]: osdmap e73: 8 total, 8 up, 8 in 2024-03-12T03:16:24.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:23 smithi151 ceph-mon[32923]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:24.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:24.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:23 smithi151 ceph-mon[32923]: from='client.34673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi094-78138-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: pgmap v65: 228 pgs: 96 unknown, 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:16:25.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: pgmap v66: 228 pgs: 96 unknown, 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:16:25.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1082297454' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2620560928' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78715-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2135651818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi094-78434-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi094-78177-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3256008021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3946638369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi094-78472-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3817618489' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3305055119' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi094-78099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1080611336' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi094-78180-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4100958568' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi094-78272-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1016017660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi094-78440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3322501849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi094-78582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2403342397' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi094-78452-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: pgmap v65: 228 pgs: 96 unknown, 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:16:25.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: pgmap v66: 228 pgs: 96 unknown, 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1082297454' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2620560928' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78715-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2135651818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi094-78434-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi094-78177-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3256008021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3946638369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi094-78472-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3817618489' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3305055119' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi094-78099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1080611336' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi094-78180-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4100958568' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi094-78272-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1016017660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi094-78440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3322501849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi094-78582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2403342397' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi094-78452-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/494097434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi094-78601-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: osdmap e74: 8 total, 8 up, 8 in 2024-03-12T03:16:25.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: Health check failed: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:25.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi094-79269-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:25.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi094-79350-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:25.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.34772 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.34838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78715-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.34718 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[40484]: from='client.34820 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/494097434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi094-78601-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: osdmap e74: 8 total, 8 up, 8 in 2024-03-12T03:16:25.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: Health check failed: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:25.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi094-79269-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:25.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi094-79350-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:25.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.34772 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.34838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78715-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.34718 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:25 smithi094 ceph-mon[29291]: from='client.34820 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: pgmap v65: 228 pgs: 96 unknown, 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:16:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: pgmap v66: 228 pgs: 96 unknown, 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:16:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1082297454' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2620560928' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78715-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2135651818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi094-78434-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi094-78177-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3256008021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3946638369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi094-78472-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3817618489' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3305055119' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi094-78099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1080611336' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi094-78180-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4100958568' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi094-78272-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1016017660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi094-78440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3322501849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi094-78582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2403342397' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi094-78452-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/494097434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi094-78601-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: osdmap e74: 8 total, 8 up, 8 in 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: Health check failed: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi094-79269-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi094-79350-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.34772 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.34838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78715-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.34718 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:25.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:25 smithi151 ceph-mon[32923]: from='client.34820 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.154 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-03-12T03:16:26.155 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [----------] Global test environment set-up. 2024-03-12T03:16:26.155 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-12T03:16:26.155 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-12T03:16:26.155 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5040 ms) 2024-03-12T03:16:26.155 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-12T03:16:26.155 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [ OK ] LibRadosService.RegisterLate (34 ms) 2024-03-12T03:16:26.155 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-12T03:16:26.155 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: cluster: 2024-03-12T03:16:26.155 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: id: 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:16:26.155 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: health: HEALTH_OK 2024-03-12T03:16:26.155 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: services: 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: data: 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: pools: 6 pools, 132 pgs 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: usage: 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: pgs: 132 active+clean 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: io: 2024-03-12T03:16:26.156 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-12T03:16:26.157 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:26.157 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:26.157 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: cluster: 2024-03-12T03:16:26.157 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: id: 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:16:26.157 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: health: HEALTH_WARN 2024-03-12T03:16:26.157 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 3 pool(s) do not have an application enabled 2024-03-12T03:16:26.157 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:26.157 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: services: 2024-03-12T03:16:26.157 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-12T03:16:26.157 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-12T03:16:26.157 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-03-12T03:16:26.157 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-12T03:16:26.157 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: data: 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: pools: 9 pools, 228 pgs 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: usage: 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: pgs: 42.105% pgs unknown 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 132 active+clean 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 96 unknown 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: io: 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: client: 1023 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:26.158 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: cluster: 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: id: 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: health: HEALTH_WARN 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 3 pool(s) do not have an application enabled 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: services: 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: data: 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: pools: 9 pools, 228 pgs 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-12T03:16:26.159 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: usage: 226 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:16:26.160 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: pgs: 42.105% pgs unknown 2024-03-12T03:16:26.160 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 132 active+clean 2024-03-12T03:16:26.160 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 96 unknown 2024-03-12T03:16:26.160 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:26.160 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: io: 2024-03-12T03:16:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: pgmap v68: 868 pgs: 576 creating+peering, 132 active+clean, 160 unknown; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.34673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi094-78138-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2499896511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi094-78828-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/630859415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78695-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/313296469' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-12T03:16:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: pgmap v70: 1092 pgs: 576 creating+peering, 132 active+clean, 384 unknown; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-12T03:16:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: osdmap e75: 8 total, 8 up, 8 in 2024-03-12T03:16:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi094-78118-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24667 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi094-78081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi094-78434-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi094-78177-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi094-78472-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24679 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi094-78099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24697 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi094-78180-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi094-79269-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi094-79350-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.34772 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.34838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78715-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.34718 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.34820 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2469088620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi094-79350-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1693340182' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi094-79269-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: osdmap e76: 8 total, 8 up, 8 in 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi094-78272-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi094-78440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24760 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi094-78582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24751 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi094-78452-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24757 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi094-78601-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24820 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi094-78828-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78695-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi094-79350-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:26.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi094-79269-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:26.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:26.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/706963393' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-12T03:16:26.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: pgmap v68: 868 pgs: 576 creating+peering, 132 active+clean, 160 unknown; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:26.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.34673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi094-78138-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2499896511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi094-78828-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/630859415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78695-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/313296469' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-12T03:16:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: pgmap v70: 1092 pgs: 576 creating+peering, 132 active+clean, 384 unknown; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-12T03:16:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: osdmap e75: 8 total, 8 up, 8 in 2024-03-12T03:16:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi094-78118-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24667 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi094-78081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi094-78434-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi094-78177-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi094-78472-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24679 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi094-78099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24697 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi094-78180-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi094-79269-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi094-79350-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.34772 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: pgmap v68: 868 pgs: 576 creating+peering, 132 active+clean, 160 unknown; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:26.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.34673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi094-78138-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2499896511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi094-78828-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/630859415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78695-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/313296469' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-12T03:16:26.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: pgmap v70: 1092 pgs: 576 creating+peering, 132 active+clean, 384 unknown; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-12T03:16:26.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: osdmap e75: 8 total, 8 up, 8 in 2024-03-12T03:16:26.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi094-78118-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24667 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi094-78081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:26.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi094-78434-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi094-78177-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi094-78472-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24679 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi094-78099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24697 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi094-78180-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi094-79269-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:26.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi094-79350-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:26.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.34772 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.34838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78715-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.34718 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.34820 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2469088620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi094-79350-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1693340182' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi094-79269-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: osdmap e76: 8 total, 8 up, 8 in 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi094-78272-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi094-78440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24760 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi094-78582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24751 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi094-78452-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24757 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi094-78601-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24820 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi094-78828-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78695-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi094-79350-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:26.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi094-79269-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:26.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:26.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/706963393' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-12T03:16:26.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.34838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78715-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.34718 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.34820 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:26.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2469088620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi094-79350-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:26.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1693340182' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi094-79269-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: osdmap e76: 8 total, 8 up, 8 in 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi094-78272-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi094-78440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24760 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi094-78582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24751 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi094-78452-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24757 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi094-78601-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24820 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi094-78828-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78695-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi094-79350-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi094-79269-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:26.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/706963393' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-12T03:16:26.819 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-12T03:16:26.819 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [----------] Global test environment set-up. 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (5 ms) 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (6 ms) 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (4 ms) 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (5 ms) 2024-03-12T03:16:26.820 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (3 ms) 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (3 ms) 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (4 ms) 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (6 ms) 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (4 ms) 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [----------] 12 tests from AsioRados (43 ms total) 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: 2024-03-12T03:16:26.821 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [----------] Global test environment tear-down 2024-03-12T03:16:26.822 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (10081 ms total) 2024-03-12T03:16:26.822 INFO:tasks.workunit.client.0.smithi094.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-12T03:16:27.103 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-12T03:16:27.103 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [----------] Global test environment set-up. 2024-03-12T03:16:27.103 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-12T03:16:27.103 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ OK ] LibRadosList.ListObjects (340 ms) 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (40 ms) 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: myset foo1,foo2,foo3 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo1 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo2 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo3 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: myset foo1,foo4,foo5 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo4 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo5 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo1 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: myset foo6,foo7 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo7 2024-03-12T03:16:27.104 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo6 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: ns1:foo4 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: ns1:foo5 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: ns2:foo7 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: ns2:foo6 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: ns1:foo1 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: :foo1 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: :foo2 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: :foo3 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (88 ms) 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 1 0 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 10 0 2024-03-12T03:16:27.105 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 13 0 2024-03-12T03:16:27.106 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 7 0 2024-03-12T03:16:27.106 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 14 0 2024-03-12T03:16:27.106 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 0 0 2024-03-12T03:16:27.106 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 15 0 2024-03-12T03:16:27.106 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 11 0 2024-03-12T03:16:27.106 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 5 0 2024-03-12T03:16:27.106 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 8 0 2024-03-12T03:16:27.106 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 6 0 2024-03-12T03:16:27.106 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 3 0 2024-03-12T03:16:27.106 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 4 0 2024-03-12T03:16:27.106 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 12 0 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 9 0 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 2 0 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (69 ms) 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: x cursor=MIN 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=1 cursor=16:02547ec2:::1:head 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=10 cursor=16:52ea6a34:::10:head 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=13 cursor=16:566253c9:::13:head 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=7 cursor=16:5c6b0b28:::7:head 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=14 cursor=16:62a1935d:::14:head 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=0 cursor=16:6cac518f:::0:head 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=15 cursor=16:863748b0:::15:head 2024-03-12T03:16:27.107 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=11 cursor=16:89d3ae78:::11:head 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=5 cursor=16:b29083e3:::5:head 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=8 cursor=16:bd63b0f1:::8:head 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=6 cursor=16:c4fdafeb:::6:head 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=3 cursor=16:cfc208b3:::3:head 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=4 cursor=16:d83876eb:::4:head 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=12 cursor=16:de5d7c5f:::12:head 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=9 cursor=16:e960b815:::9:head 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > oid=2 cursor=16:f905c69b:::2:head 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=1 cursor=16:02547ec2:::1:head 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:02547ec2:::1:head 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:02547ec2:::1:head -> 1 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=10 cursor=16:52ea6a34:::10:head 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:52ea6a34:::10:head 2024-03-12T03:16:27.108 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:52ea6a34:::10:head -> 10 2024-03-12T03:16:27.109 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=13 cursor=16:566253c9:::13:head 2024-03-12T03:16:27.109 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:566253c9:::13:head 2024-03-12T03:16:27.109 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:566253c9:::13:head -> 13 2024-03-12T03:16:27.109 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=7 cursor=16:5c6b0b28:::7:head 2024-03-12T03:16:27.109 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:5c6b0b28:::7:head 2024-03-12T03:16:27.109 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:5c6b0b28:::7:head -> 7 2024-03-12T03:16:27.196 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=14 cursor=16:62a1935d:::14:head 2024-03-12T03:16:27.196 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:62a1935d:::14:head 2024-03-12T03:16:27.196 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:62a1935d:::14:head -> 14 2024-03-12T03:16:27.196 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=0 cursor=16:6cac518f:::0:head 2024-03-12T03:16:27.196 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:6cac518f:::0:head 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:6cac518f:::0:head -> 0 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=15 cursor=16:863748b0:::15:head 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:863748b0:::15:head 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:863748b0:::15:head -> 15 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=11 cursor=16:89d3ae78:::11:head 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:89d3ae78:::11:head 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:89d3ae78:::11:head -> 11 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=5 cursor=16:b29083e3:::5:head 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:b29083e3:::5:head 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:b29083e3:::5:head -> 5 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=8 cursor=16:bd63b0f1:::8:head 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:bd63b0f1:::8:head 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:bd63b0f1:::8:head -> 8 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=6 cursor=16:c4fdafeb:::6:head 2024-03-12T03:16:27.197 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:c4fdafeb:::6:head 2024-03-12T03:16:27.198 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:c4fdafeb:::6:head -> 6 2024-03-12T03:16:27.198 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=3 cursor=16:cfc208b3:::3:head 2024-03-12T03:16:27.198 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:cfc208b3:::3:head 2024-03-12T03:16:27.198 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:cfc208b3:::3:head -> 3 2024-03-12T03:16:27.198 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=4 cursor=16:d83876eb:::4:head 2024-03-12T03:16:27.198 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:d83876eb:::4:head 2024-03-12T03:16:27.198 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:d83876eb:::4:head -> 4 2024-03-12T03:16:27.198 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=12 cursor=16:de5d7c5f:::12:head 2024-03-12T03:16:27.198 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:de5d7c5f:::12:head 2024-03-12T03:16:27.198 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:de5d7c5f:::12:head -> 12 2024-03-12T03:16:27.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=9 cursor=16:e960b815:::9:head 2024-03-12T03:16:27.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:e960b815:::9:head 2024-03-12T03:16:27.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:e960b815:::9:head -> 9 2024-03-12T03:16:27.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : oid=2 cursor=16:f905c69b:::2:head 2024-03-12T03:16:27.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:f905c69b:::2:head 2024-03-12T03:16:27.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:f905c69b:::2:head -> 2 2024-03-12T03:16:27.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:863748b0:::15:head 2024-03-12T03:16:27.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:863748b0:::15:head expected=16:863748b0:::15:head 2024-03-12T03:16:27.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:863748b0:::15:head -> 15 2024-03-12T03:16:27.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=15 expected=15 2024-03-12T03:16:27.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:6cac518f:::0:head 2024-03-12T03:16:27.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:6cac518f:::0:head expected=16:6cac518f:::0:head 2024-03-12T03:16:27.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:6cac518f:::0:head -> 0 2024-03-12T03:16:27.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=0 expected=0 2024-03-12T03:16:27.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:62a1935d:::14:head 2024-03-12T03:16:27.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:62a1935d:::14:head expected=16:62a1935d:::14:head 2024-03-12T03:16:27.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:62a1935d:::14:head -> 14 2024-03-12T03:16:27.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=14 expected=14 2024-03-12T03:16:27.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:5c6b0b28:::7:head 2024-03-12T03:16:27.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:5c6b0b28:::7:head expected=16:5c6b0b28:::7:head 2024-03-12T03:16:27.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:5c6b0b28:::7:head -> 7 2024-03-12T03:16:27.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=7 expected=7 2024-03-12T03:16:27.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:566253c9:::13:head 2024-03-12T03:16:27.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:566253c9:::13:head expected=16:566253c9:::13:head 2024-03-12T03:16:27.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:566253c9:::13:head -> 13 2024-03-12T03:16:27.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=13 expected=13 2024-03-12T03:16:27.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:52ea6a34:::10:head 2024-03-12T03:16:27.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:52ea6a34:::10:head expected=16:52ea6a34:::10:head 2024-03-12T03:16:27.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:52ea6a34:::10:head -> 10 2024-03-12T03:16:27.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=10 expected=10 2024-03-12T03:16:27.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:f905c69b:::2:head 2024-03-12T03:16:27.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:f905c69b:::2:head expected=16:f905c69b:::2:head 2024-03-12T03:16:27.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:f905c69b:::2:head -> 2 2024-03-12T03:16:27.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=2 expected=2 2024-03-12T03:16:27.202 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:e960b815:::9:head 2024-03-12T03:16:27.202 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:e960b815:::9:head expected=16:e960b815:::9:head 2024-03-12T03:16:27.202 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:e960b815:::9:head -> 9 2024-03-12T03:16:27.347 INFO:tasks.workunit.client.0.smithi094.stdout: api_li list_parallel: process_1_[78865]: starting. 2024-03-12T03:16:27.347 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_1_[78865]: creating pool ceph_test_rados_list_parallel.smithi094-78824 2024-03-12T03:16:27.347 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_1_[78865]: created object 0... 2024-03-12T03:16:27.347 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_1_[78865]: created object 25... 2024-03-12T03:16:27.347 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_1_[78865]: created object 49... 2024-03-12T03:16:27.348 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_1_[78865]: finishing. 2024-03-12T03:16:27.348 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_1_[78865]: shutting down. 2024-03-12T03:16:27.348 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_2_[78866]: starting. 2024-03-12T03:16:27.348 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_2_[78866]: listing objects. 2024-03-12T03:16:27.348 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_2_[78866]: listed object 0... 2024-03-12T03:16:27.348 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_2_[78866]: listed object 25... 2024-03-12T03:16:27.348 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_2_[78866]: saw 50 objects 2024-03-12T03:16:27.348 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_2_[78866]: shutting down. 2024-03-12T03:16:27.348 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.348 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_3_[80067]: starting. 2024-03-12T03:16:27.348 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_3_[80067]: creating pool ceph_test_rados_list_parallel.smithi094-78824 2024-03-12T03:16:27.348 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_3_[80067]: created object 0... 2024-03-12T03:16:27.348 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_3_[80067]: created object 25... 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_3_[80067]: created object 49... 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_3_[80067]: finishing. 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_3_[80067]: shutting down. 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_4_[80068]: starting. 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_4_[80068]: listing objects. 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_4_[80068]: listed object 0... 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_4_[80068]: listed object 25... 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_4_[80068]: saw 43 objects 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_4_[80068]: shutting down. 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_5_[80069]: starting. 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_5_[80069]: removed 25 objects... 2024-03-12T03:16:27.349 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_5_[80069]: removed half of the objects 2024-03-12T03:16:27.350 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_5_[80069]: removed 50 objects... 2024-03-12T03:16:27.350 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_5_[80069]: removed 50 objects 2024-03-12T03:16:27.350 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_5_[80069]: shutting down. 2024-03-12T03:16:27.350 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.350 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.350 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_6_[80115]: starting. 2024-03-12T03:16:27.350 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_6_[80115]: creating pool ceph_test_rados_list_parallel.smithi094-78824 2024-03-12T03:16:27.350 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_6_[80115]: created object 0... 2024-03-12T03:16:27.350 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_6_[80115]: created object 25... 2024-03-12T03:16:27.350 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_6_[80115]: created object 49... 2024-03-12T03:16:27.350 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_6_[80115]: finishing. 2024-03-12T03:16:27.350 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_6_[80115]: shutting down. 2024-03-12T03:16:27.350 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_7_[80116]: starting. 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_7_[80116]: listing objects. 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_7_[80116]: listed object 0... 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_7_[80116]: listed object 25... 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_7_[80116]: listed object 50... 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_7_[80116]: saw 56 objects 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_7_[80116]: shutting down. 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_8_[80117]: starting. 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_8_[80117]: added 25 objects... 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_8_[80117]: added half of the objects 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_8_[80117]: added 50 objects... 2024-03-12T03:16:27.351 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_8_[80117]: added 50 objects 2024-03-12T03:16:27.352 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_8_[80117]: shutting down. 2024-03-12T03:16:27.352 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.352 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.546 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.546 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_9_[80206]: starting. 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_9_[80206]: creating pool ceph_test_rados_list_parallel.smithi094-78824 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_9_[80206]: created object 0... 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_9_[80206]: created object 25... 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_9_[80206]: created object 49... 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_9_[80206]: finishing. 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_9_[80206]: shutting down. 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_10_[80207]: starting. 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_10_[80207]: listing objects. 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_10_[80207]: listed object 0... 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_10_[80207]: listed object 25... 2024-03-12T03:16:27.547 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_10_[80207]: listed object 50... 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_10_[80207]: listed object 75... 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_10_[80207]: saw 97 objects 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_10_[80207]: shutting down. 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_13_[80210]: starting. 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_13_[80210]: removed 25 objects... 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_13_[80210]: removed half of the objects 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_13_[80210]: removed 50 objects... 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_13_[80210]: removed 50 objects 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_13_[80210]: shutting down. 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.548 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_11_[80208]: starting. 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_11_[80208]: added 25 objects... 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_11_[80208]: added half of the objects 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_11_[80208]: added 50 objects... 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_11_[80208]: added 50 objects 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_11_[80208]: shutting down. 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_12_[80209]: starting. 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_12_[80209]: added 25 objects... 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_12_[80209]: added half of the objects 2024-03-12T03:16:27.549 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_12_[80209]: added 50 objects... 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_12_[80209]: added 50 objects 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_12_[80209]: shutting down. 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_14_[80286]: starting. 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_14_[80286]: creating pool ceph_test_rados_list_parallel.smithi094-78824 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_14_[80286]: created object 0... 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_14_[80286]: created object 25... 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_14_[80286]: created object 49... 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_14_[80286]: finishing. 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_14_[80286]: shutting down. 2024-03-12T03:16:27.550 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.551 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.551 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.551 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:27.551 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_15_[80287]: starting. 2024-03-12T03:16:27.551 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_15_[80287]: listing objects. 2024-03-12T03:16:27.551 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_15_[80287]: listed object 0... 2024-03-12T03:16:27.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:16:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi094-78118-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24667 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi094-78081-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi094-78434-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi094-78177-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi094-78472-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24679 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi094-78099-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24697 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi094-78180-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi094-78272-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi094-78440-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24760 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi094-78582-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24751 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi094-78452-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24757 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi094-78601-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi094-78828-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78695-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:27.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi094-78644-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:27.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: osdmap e77: 8 total, 8 up, 8 in 2024-03-12T03:16:27.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi094-78644-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:27.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:28.049 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_15_[80287]: listed object 2 api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-12T03:16:28.050 INFO:tasks.workunit.client.0.smithi094.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-12T03:16:28.050 INFO:tasks.workunit.client.0.smithi094.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-12T03:16:28.050 INFO:tasks.workunit.client.0.smithi094.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-12T03:16:28.050 INFO:tasks.workunit.client.0.smithi094.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (11103 ms) 2024-03-12T03:16:28.050 INFO:tasks.workunit.client.0.smithi094.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (11103 ms total) 2024-03-12T03:16:28.050 INFO:tasks.workunit.client.0.smithi094.stdout: api_cls_remote_reads: 2024-03-12T03:16:28.050 INFO:tasks.workunit.client.0.smithi094.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-12T03:16:28.050 INFO:tasks.workunit.client.0.smithi094.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (11103 ms total) 2024-03-12T03:16:28.050 INFO:tasks.workunit.client.0.smithi094.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-12T03:16:28.050 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (71 ms) 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (63 ms) 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (11040 ms) 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (11174 ms total) 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (11175 ms total) 2024-03-12T03:16:28.051 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-12T03:16:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi094-78118-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24667 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi094-78081-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi094-78434-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi094-78177-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi094-78472-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24679 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi094-78099-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24697 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi094-78180-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi094-78272-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi094-78440-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24760 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi094-78582-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24751 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi094-78452-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24757 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi094-78601-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi094-78828-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78695-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:28.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi094-78644-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:28.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi094-78118-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24667 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi094-78081-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi094-78434-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi094-78177-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi094-78472-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24679 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi094-78099-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24697 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi094-78180-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi094-78272-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi094-78440-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24760 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi094-78582-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24751 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi094-78452-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24757 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi094-78601-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi094-78828-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78695-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:28.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi094-78644-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:28.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: osdmap e77: 8 total, 8 up, 8 in 2024-03-12T03:16:28.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi094-78644-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:28.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:28.213 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: osdmap e77: 8 total, 8 up, 8 in 2024-03-12T03:16:28.213 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi094-78644-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:28.213 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:29.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[29291]: pgmap v73: 1188 pgs: 552 creating+peering, 132 active+clean, 504 unknown; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:29.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[29291]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi094-79350-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi094-79350-1"}]': finished 2024-03-12T03:16:29.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[29291]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi094-79269-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi094-79269-1"}]': finished 2024-03-12T03:16:29.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:29.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1185459947' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-12T03:16:29.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:29.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[29291]: osdmap e78: 8 total, 8 up, 8 in 2024-03-12T03:16:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[40484]: pgmap v73: 1188 pgs: 552 creating+peering, 132 active+clean, 504 unknown; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[40484]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi094-79350-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi094-79350-1"}]': finished 2024-03-12T03:16:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[40484]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi094-79269-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi094-79269-1"}]': finished 2024-03-12T03:16:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:29.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1185459947' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-12T03:16:29.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:29.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:28 smithi094 ceph-mon[40484]: osdmap e78: 8 total, 8 up, 8 in 2024-03-12T03:16:29.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:28 smithi151 ceph-mon[32923]: pgmap v73: 1188 pgs: 552 creating+peering, 132 active+clean, 504 unknown; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:16:29.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:28 smithi151 ceph-mon[32923]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi094-79350-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi094-79350-1"}]': finished 2024-03-12T03:16:29.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:28 smithi151 ceph-mon[32923]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi094-79269-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi094-79269-1"}]': finished 2024-03-12T03:16:29.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:29.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1185459947' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-12T03:16:29.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:29.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:28 smithi151 ceph-mon[32923]: osdmap e78: 8 total, 8 up, 8 in 2024-03-12T03:16:29.891 INFO:tasks.workunit.client.0.smithi094.stdout:5... 2024-03-12T03:16:29.891 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_15_[80287]: listed object 50... 2024-03-12T03:16:29.891 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_15_[80287]: listed object 75... 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_15_[80287]: listed object 100... 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_15_[80287]: listed object 125... 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_15_[80287]: saw 150 objects 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_15_[80287]: shutting down. 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_16_[80288]: starting. 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_16_[80288]: added 25 objects... 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_16_[80288]: added half of the objects 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_16_[80288]: added 50 objects... 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_16_[80288]: added 50 objects 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: process_16_[80288]: shutting down. 2024-03-12T03:16:29.892 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:29.893 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:29.893 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:29.893 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:29.893 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******************************* 2024-03-12T03:16:29.893 INFO:tasks.workunit.client.0.smithi094.stdout: list_parallel: ******* SUCCESS ********** 2024-03-12T03:16:29.893 INFO:tasks.workunit.client.0.smithi094.stdout: handler_error: Running main() from gmock_main.cc 2024-03-12T03:16:29.893 INFO:tasks.workunit.client.0.smithi094.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-03-12T03:16:29.893 INFO:tasks.workunit.client.0.smithi094.stdout: handler_error: [----------] Global test environment set-up. 2024-03-12T03:16:29.893 INFO:tasks.workunit.client.0.smithi094.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-03-12T03:16:29.894 INFO:tasks.workunit.client.0.smithi094.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-03-12T03:16:29.894 INFO:tasks.workunit.client.0.smithi094.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (12851 ms) 2024-03-12T03:16:29.894 INFO:tasks.workunit.client.0.smithi094.stdout: handler_error: [----------] 1 test from neocls_handler_error (12851 ms total) 2024-03-12T03:16:29.894 INFO:tasks.workunit.client.0.smithi094.stdout: handler_error: 2024-03-12T03:16:29.894 INFO:tasks.workunit.client.0.smithi094.stdout: handler_error: [----------] Global test environment tear-down 2024-03-12T03:16:29.894 INFO:tasks.workunit.client.0.smithi094.stdout: handler_error: [==========] 1 test from 1 test suite ran. (12851 ms total) 2024-03-12T03:16:29.894 INFO:tasks.workunit.client.0.smithi094.stdout: handler_error: [ PASSED ] 1 test. 2024-03-12T03:16:29.911 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-03-12T03:16:29.911 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-12T03:16:29.911 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-12T03:16:29.911 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (233 ms) 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (71 ms) 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (16 ms) 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (30 ms) 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (7 ms) 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (8 ms) 2024-03-12T03:16:29.912 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (5 ms) 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (10 ms) 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5 ms) 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (5 ms) 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (4 ms) 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (6 ms) 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (14 ms) 2024-03-12T03:16:29.913 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-12T03:16:29.914 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (14 ms) 2024-03-12T03:16:29.914 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-12T03:16:29.914 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (8 ms) 2024-03-12T03:16:29.914 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-12T03:16:29.914 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-03-12T03:16:29.914 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (440 ms total) 2024-03-12T03:16:29.914 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: 2024-03-12T03:16:29.914 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-12T03:16:29.914 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (12982 ms total) 2024-03-12T03:16:29.914 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-12T03:16:29.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: Running main() from gmock_main.cc 2024-03-12T03:16:29.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-12T03:16:29.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-12T03:16:29.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-12T03:16:29.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-12T03:16:29.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (59 ms) 2024-03-12T03:16:29.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-12T03:16:29.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (55 ms) 2024-03-12T03:16:29.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-12T03:16:29.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (12911 ms) 2024-03-12T03:16:29.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-12T03:16:29.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:28.854670+0000 mon.a [INF] from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi094-78644-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]': finished 2024-03-12T03:16:29.998 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:28.868150+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/2305176016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:29.998 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:28.868434+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/2880510066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:29.998 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.135659+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-03-12T03:16:29.998 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.136268+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-03-12T03:16:29.998 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.136725+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-03-12T03:16:29.998 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.137104+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-03-12T03:16:29.998 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.137504+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-03-12T03:16:29.998 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.137917+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-03-12T03:16:29.998 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.138315+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-03-12T03:16:29.998 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.138724+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-03-12T03:16:29.998 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.139100+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-03-12T03:16:29.998 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.139498+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-03-12T03:16:29.998 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.140856+0000 mon.a [INF] from='client.35105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi094-78644-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]': finished 2024-03-12T03:16:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2305176016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2880510066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: osdmap e79: 8 total, 8 up, 8 in 2024-03-12T03:16:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-03-12T03:16:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-03-12T03:16:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-03-12T03:16:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-03-12T03:16:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-03-12T03:16:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-03-12T03:16:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-03-12T03:16:30.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-03-12T03:16:30.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-03-12T03:16:30.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-03-12T03:16:30.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.35105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:30.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[40484]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:30.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:30.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi094-78644-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]': finished 2024-03-12T03:16:30.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2305176016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2880510066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: osdmap e79: 8 total, 8 up, 8 in 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.35105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:30.213 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:29 smithi094 ceph-mon[29291]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:30.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:30.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi094-78644-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]': finished 2024-03-12T03:16:30.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2305176016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2880510066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: osdmap e79: 8 total, 8 up, 8 in 2024-03-12T03:16:30.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-03-12T03:16:30.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-03-12T03:16:30.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-03-12T03:16:30.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-03-12T03:16:30.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-03-12T03:16:30.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-03-12T03:16:30.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-03-12T03:16:30.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-03-12T03:16:30.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-03-12T03:16:30.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-03-12T03:16:30.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.35105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:30.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:29 smithi151 ceph-mon[32923]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:30.922 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:16:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:16:30.924 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.141322+0000 mon.a [INF] from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.924 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.382196+0000 mon.a [WRN] Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:30.924 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.874358+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:30.924 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.879543+0000 mon.a [INF] from='client.35105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:30.924 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.879613+0000 mon.a [INF] from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:30.924 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.887107+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/1693340182' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:30.924 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.888480+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/433112286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi094-78081-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.925 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.892584+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:30.925 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.985938+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:30.925 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.992568+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-12T03:16:30.925 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:29.992838+0000 client.admin [INF] onexx 2024-03-12T03:16:30.925 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.000048+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:30.925 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.014853+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:30.925 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.194050+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:30.925 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.432986+0000 mon.a [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-12T03:16:30.925 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.433258+0000 mon.a [INF] from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:30.925 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.433755+0000 mon.a [INF] from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:30.925 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.433883+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: pgmap v75: 1012 pgs: 96 creating+peering, 836 active+clean, 80 unknown; 623 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 144 KiB/s wr, 388 op/s 2024-03-12T03:16:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-03-12T03:16:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-03-12T03:16:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-03-12T03:16:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-03-12T03:16:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-03-12T03:16:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-03-12T03:16:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-03-12T03:16:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-03-12T03:16:31.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-03-12T03:16:31.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-03-12T03:16:31.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: 20.0 deep-scrub starts 2024-03-12T03:16:31.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: 20.0 deep-scrub ok 2024-03-12T03:16:31.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: 20.3 deep-scrub starts 2024-03-12T03:16:31.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: 20.3 deep-scrub ok 2024-03-12T03:16:31.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:31.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.35105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:31.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:31.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1693340182' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:31.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/433112286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi094-78081-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:31.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:31.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:31.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-12T03:16:31.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: onexx 2024-03-12T03:16:31.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: pgmap v75: 1012 pgs: 96 creating+peering, 836 active+clean, 80 unknown; 623 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 144 KiB/s wr, 388 op/s 2024-03-12T03:16:31.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-03-12T03:16:31.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-03-12T03:16:31.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-03-12T03:16:31.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-03-12T03:16:31.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-03-12T03:16:31.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: 20.0 deep-scrub starts 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: 20.0 deep-scrub ok 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: 20.3 deep-scrub starts 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: 20.3 deep-scrub ok 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.35105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1693340182' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/433112286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi094-78081-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:31.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:31.214 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-12T03:16:31.214 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: onexx 2024-03-12T03:16:31.214 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:31.214 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:31.214 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: osdmap e80: 8 total, 8 up, 8 in 2024-03-12T03:16:31.214 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:31.214 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:31.214 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-12T03:16:31.214 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:31.215 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:31.215 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:31.215 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:31.215 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:31.215 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:31.215 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi094-78472-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:31.215 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:31.215 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi094-78472-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:31.215 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:31.215 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:31.215 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.25225 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi094-78081-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:31.215 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:31.216 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:31.216 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:31.216 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: osdmap e80: 8 total, 8 up, 8 in 2024-03-12T03:16:31.216 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:31.216 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:31.216 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-12T03:16:31.216 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:31.216 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:31.216 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:31.216 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:31.216 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:31.217 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:31.217 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi094-78472-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:31.217 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:31.217 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi094-78472-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:31.217 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:31.217 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:31.217 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.25225 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi094-78081-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:31.217 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:30 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:31.218 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: pgmap v75: 1012 pgs: 96 creating+peering, 836 active+clean, 80 unknown; 623 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 144 KiB/s wr, 388 op/s 2024-03-12T03:16:31.218 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-03-12T03:16:31.218 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-03-12T03:16:31.218 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-03-12T03:16:31.218 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-03-12T03:16:31.218 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-03-12T03:16:31.218 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-03-12T03:16:31.218 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-03-12T03:16:31.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-03-12T03:16:31.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-03-12T03:16:31.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-03-12T03:16:31.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: 20.0 deep-scrub starts 2024-03-12T03:16:31.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: 20.0 deep-scrub ok 2024-03-12T03:16:31.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: 20.3 deep-scrub starts 2024-03-12T03:16:31.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: 20.3 deep-scrub ok 2024-03-12T03:16:31.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:31.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.35105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:31.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:31.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1693340182' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:31.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/433112286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi094-78081-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:31.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: onexx 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: osdmap e80: 8 total, 8 up, 8 in 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:31.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:31.221 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:31.221 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi094-78472-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:31.221 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:31.221 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi094-78472-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:31.221 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:31.221 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:31.221 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.25225 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi094-78081-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:31.221 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:30 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:32.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[29291]: 20.8 deep-scrub starts 2024-03-12T03:16:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[29291]: 20.8 deep-scrub ok 2024-03-12T03:16:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[29291]: 20.2 deep-scrub starts 2024-03-12T03:16:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[29291]: 20.2 deep-scrub ok 2024-03-12T03:16:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[29291]: from='client.35237 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-12T03:16:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[29291]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[29291]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[29291]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi094-78472-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:32.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[40484]: 20.8 deep-scrub starts 2024-03-12T03:16:32.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[40484]: 20.8 deep-scrub ok 2024-03-12T03:16:32.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[40484]: 20.2 deep-scrub starts 2024-03-12T03:16:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[40484]: 20.2 deep-scrub ok 2024-03-12T03:16:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[40484]: from='client.35237 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-12T03:16:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[40484]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[40484]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:32 smithi094 ceph-mon[40484]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi094-78472-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:32.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:32 smithi151 ceph-mon[32923]: 20.8 deep-scrub starts 2024-03-12T03:16:32.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:32 smithi151 ceph-mon[32923]: 20.8 deep-scrub ok 2024-03-12T03:16:32.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:32 smithi151 ceph-mon[32923]: 20.2 deep-scrub starts 2024-03-12T03:16:32.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:32 smithi151 ceph-mon[32923]: 20.2 deep-scrub ok 2024-03-12T03:16:32.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:32 smithi151 ceph-mon[32923]: from='client.35237 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-12T03:16:32.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:32 smithi151 ceph-mon[32923]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:32.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:32 smithi151 ceph-mon[32923]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:32.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:32 smithi151 ceph-mon[32923]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi094-78472-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: pgmap v78: 756 pgs: 96 creating+peering, 420 active+clean, 240 unknown; 586 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 21 KiB/s wr, 236 op/s 2024-03-12T03:16:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-12T03:16:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi094-79269-1"}]': finished 2024-03-12T03:16:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.25225 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi094-78081-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi094-78472-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi094-78582-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: twoxx 2024-03-12T03:16:33.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1243497563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:33.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1693340182' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:33.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2880510066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:33.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:33.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:33.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: 20.5 deep-scrub starts 2024-03-12T03:16:33.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: 20.5 deep-scrub ok 2024-03-12T03:16:33.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:33.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:33.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: 20.6 deep-scrub starts 2024-03-12T03:16:33.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: 20.6 deep-scrub ok 2024-03-12T03:16:33.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: osdmap e81: 8 total, 8 up, 8 in 2024-03-12T03:16:33.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: pgmap v78: 756 pgs: 96 creating+peering, 420 active+clean, 240 unknown; 586 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 21 KiB/s wr, 236 op/s 2024-03-12T03:16:33.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-12T03:16:33.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:33.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi094-79269-1"}]': finished 2024-03-12T03:16:33.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.25225 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi094-78081-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:33.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:33.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi094-78472-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:33.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi094-78582-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:33.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: twoxx 2024-03-12T03:16:33.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1243497563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:33.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1693340182' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:33.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2880510066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:33.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:33.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:33.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: 20.5 deep-scrub starts 2024-03-12T03:16:33.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: 20.5 deep-scrub ok 2024-03-12T03:16:33.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:33.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:33.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: 20.6 deep-scrub starts 2024-03-12T03:16:33.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: 20.6 deep-scrub ok 2024-03-12T03:16:33.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: osdmap e81: 8 total, 8 up, 8 in 2024-03-12T03:16:33.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:33.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-12T03:16:33.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.25237 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:33.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi094-78472-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:33.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:33.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi094-78582-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:33.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:33.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: osdmap e82: 8 total, 8 up, 8 in 2024-03-12T03:16:33.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:33.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2469088620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:33.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:33.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:33.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:33.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:33.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:33.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:33.716 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:16:33.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:33.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-12T03:16:33.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.25237 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:33.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi094-78472-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:33.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:33.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi094-78582-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:33.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:33.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: osdmap e82: 8 total, 8 up, 8 in 2024-03-12T03:16:33.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:33.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2469088620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:33.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:33.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:33.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:33.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:33.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:33.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:33.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:33 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:16:33.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: pgmap v78: 756 pgs: 96 creating+peering, 420 active+clean, 240 unknown; 586 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 21 KiB/s wr, 236 op/s 2024-03-12T03:16:33.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-12T03:16:33.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:33.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi094-79269-1"}]': finished 2024-03-12T03:16:33.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.25225 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi094-78081-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:33.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:33.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi094-78472-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi094-78582-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: twoxx 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1243497563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1693340182' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2880510066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: 20.5 deep-scrub starts 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: 20.5 deep-scrub ok 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: 20.6 deep-scrub starts 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: 20.6 deep-scrub ok 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: osdmap e81: 8 total, 8 up, 8 in 2024-03-12T03:16:33.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.25237 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi094-78472-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi094-78582-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: osdmap e82: 8 total, 8 up, 8 in 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2469088620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:33.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:33.721 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:33 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:16:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: 20.1 deep-scrub starts 2024-03-12T03:16:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: 20.1 deep-scrub ok 2024-03-12T03:16:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: 20.7 deep-scrub starts 2024-03-12T03:16:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: 20.7 deep-scrub ok 2024-03-12T03:16:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: 20.9 deep-scrub starts 2024-03-12T03:16:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: 20.9 deep-scrub ok 2024-03-12T03:16:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: pgmap v81: 836 pgs: 288 creating+peering, 548 active+clean; 145 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 127 op/s 2024-03-12T03:16:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: 20.4 deep-scrub starts 2024-03-12T03:16:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: 20.4 deep-scrub ok 2024-03-12T03:16:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.35237 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-12T03:16:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi094-79269-1"}]': finished 2024-03-12T03:16:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2880510066' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi094-78099-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-12T03:16:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi094-79269-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: osdmap e83: 8 total, 8 up, 8 in 2024-03-12T03:16:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/854350452' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi094-78180-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: 20.1 deep-scrub starts 2024-03-12T03:16:34.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: 20.1 deep-scrub ok 2024-03-12T03:16:34.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: 20.7 deep-scrub starts 2024-03-12T03:16:34.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: 20.7 deep-scrub ok 2024-03-12T03:16:34.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: 20.9 deep-scrub starts 2024-03-12T03:16:34.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: 20.9 deep-scrub ok 2024-03-12T03:16:34.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: pgmap v81: 836 pgs: 288 creating+peering, 548 active+clean; 145 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 127 op/s 2024-03-12T03:16:34.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: 20.4 deep-scrub starts 2024-03-12T03:16:34.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: 20.4 deep-scrub ok 2024-03-12T03:16:34.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:34.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.35237 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-12T03:16:34.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:34.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi094-79269-1"}]': finished 2024-03-12T03:16:34.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:34.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2880510066' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi094-78099-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-12T03:16:34.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi094-79269-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: osdmap e83: 8 total, 8 up, 8 in 2024-03-12T03:16:34.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/854350452' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:34.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:34.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi094-78180-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi094-78099-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-12T03:16:34.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:34.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi094-78118-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:34.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:34.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi094-78180-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi094-79269-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.25360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi094-78118-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:34.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi094-78099-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-12T03:16:34.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:34.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi094-78118-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:34.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:34.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi094-78180-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi094-79269-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.25360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi094-78118-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:34.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: 20.1 deep-scrub starts 2024-03-12T03:16:34.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: 20.1 deep-scrub ok 2024-03-12T03:16:34.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: 20.7 deep-scrub starts 2024-03-12T03:16:34.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: 20.7 deep-scrub ok 2024-03-12T03:16:34.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: 20.9 deep-scrub starts 2024-03-12T03:16:34.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: 20.9 deep-scrub ok 2024-03-12T03:16:34.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: pgmap v81: 836 pgs: 288 creating+peering, 548 active+clean; 145 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 127 op/s 2024-03-12T03:16:34.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: 20.4 deep-scrub starts 2024-03-12T03:16:34.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: 20.4 deep-scrub ok 2024-03-12T03:16:34.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:34.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.35237 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-12T03:16:34.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:34.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi094-79269-1"}]': finished 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2880510066' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi094-78099-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi094-79269-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: osdmap e83: 8 total, 8 up, 8 in 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/854350452' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi094-78180-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi094-78099-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:34.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi094-78118-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:34.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:34.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi094-78180-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi094-79269-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.25360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:34.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi094-78118-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:34.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:35.415 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.434299+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:35.415 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.674450+0000 mon.a [INF] from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:35.415 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.674707+0000 mon.a [INF] from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:35.415 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.675018+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi094-78472-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:35.415 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.675199+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:35.415 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.676652+0000 mon.a [INF] from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi094-78472-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:35.416 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.677076+0000 mon.a [INF] from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:35.416 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.681389+0000 mon.a [INF] from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi094-79269-1"}]: dispatch 2024-03-12T03:16:35.416 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.681774+0000 mon.a [INF] from='client.25225 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi094-78081-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:35.416 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:30.682166+0000 mon.a [INF] from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:35.416 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.409563+0000 mon.a [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-12T03:16:35.416 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.415141+0000 mon.a [INF] from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.416 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.415211+0000 mon.a [INF] from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi094-79269-1"}]': finished 2024-03-12T03:16:35.416 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.415262+0000 mon.a [INF] from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi094-78099-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.416 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.424071+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/2880510066' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi094-78099-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-12T03:16:35.416 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.461368+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi094-79269-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:35.416 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.926251+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:35.416 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.926711+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:35.416 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.926960+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/854350452' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:35.417 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.929271+0000 mon.a [INF] from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:35.417 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.929722+0000 mon.a [INF] from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:35.417 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.929924+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi094-78180-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:35.417 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.930522+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:35.417 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.933307+0000 mon.a [INF] from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi094-78099-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-12T03:16:35.417 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.934707+0000 mon.a [INF] from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:35.417 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.935177+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:35.417 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.935485+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi094-78118-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:35.417 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.935934+0000 mon.a [INF] from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:35.417 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.936473+0000 mon.a [INF] from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:35.417 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.943152+0000 mon.a [INF] from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi094-78180-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:35.417 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.943491+0000 mon.a [INF] from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:35.492 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.946875+0000 mon.a [INF] from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi094-79269-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:35.493 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.950716+0000 mon.a [INF] from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:35.493 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.950956+0000 mon.a [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:35.493 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.951124+0000 mon.a [INF] from='client.25360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:35.493 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:33.951356+0000 mon.a [INF] from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi094-78118-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:35.493 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.416813+0000 mon.a [WRN] Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:35.493 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.424143+0000 mon.a [INF] from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi094-78472-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi094-78472-7"}]': finished 2024-03-12T03:16:35.493 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.424225+0000 mon.a [INF] from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi094-78582-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi094-78582-7"}]': finished 2024-03-12T03:16:35.493 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.424289+0000 mon.a [INF] from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi094-78099-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-12T03:16:35.493 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.424335+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.493 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.424383+0000 mon.a [INF] from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]': finished 2024-03-12T03:16:35.494 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.424425+0000 mon.a [INF] from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi094-79350-1"}]': finished 2024-03-12T03:16:35.494 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.424475+0000 mon.a [INF] from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi094-78180-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.494 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.424539+0000 mon.a [INF] from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.494 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.424589+0000 mon.a [INF] from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi094-79269-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.494 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.424669+0000 mon.a [INF] from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.494 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.424763+0000 mon.a [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.494 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.424855+0000 mon.a [INF] from='client.25360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.494 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.424944+0000 mon.a [INF] from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi094-78118-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.494 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.429530+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi094-79269-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:35.494 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.429758+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi094-78138-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:35.494 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.429904+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi094-78180-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:35.494 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.430014+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:35.494 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.430490+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi094-78272-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:35.495 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.432217+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/2469088620' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:35.495 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.432570+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi094-78118-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:35.495 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.441625+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:35.495 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.454949+0000 mon.a [INF] from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi094-79269-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi094-78472-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi094-78472-7"}]': finished 2024-03-12T03:16:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi094-78582-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi094-78582-7"}]': finished 2024-03-12T03:16:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi094-78099-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-12T03:16:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]': finished 2024-03-12T03:16:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi094-79350-1"}]': finished 2024-03-12T03:16:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi094-78180-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi094-79269-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.25360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi094-78118-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi094-79269-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:35.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi094-78138-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:35.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: osdmap e84: 8 total, 8 up, 8 in 2024-03-12T03:16:35.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi094-78180-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:35.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:35.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi094-78272-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:35.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2469088620' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:35.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi094-78118-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:35.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:35.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi094-79269-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:35.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi094-78180-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:35.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi094-78138-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:35.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi094-78272-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:35.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:35.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:35.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi094-78118-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:35.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:35.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-12T03:16:35.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:35.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:35.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi094-78472-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi094-78472-7"}]': finished 2024-03-12T03:16:35.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi094-78582-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi094-78582-7"}]': finished 2024-03-12T03:16:35.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi094-78099-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-12T03:16:35.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]': finished 2024-03-12T03:16:35.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi094-79350-1"}]': finished 2024-03-12T03:16:35.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi094-78180-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi094-79269-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.25360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi094-78118-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi094-79269-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:35.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi094-78138-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:35.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: osdmap e84: 8 total, 8 up, 8 in 2024-03-12T03:16:35.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi094-78180-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:35.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:35.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi094-78272-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:35.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2469088620' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:35.965 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi094-78118-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:35.965 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:35.965 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi094-79269-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:35.965 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi094-78180-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:35.965 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi094-78138-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:35.965 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi094-78272-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:35.965 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:35.965 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:35.965 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi094-78118-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:35.965 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:35.965 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-12T03:16:35.965 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:35.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:35.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi094-78472-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi094-78472-7"}]': finished 2024-03-12T03:16:35.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi094-78582-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi094-78582-7"}]': finished 2024-03-12T03:16:35.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi094-78099-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-12T03:16:35.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi094-78644-1"}]': finished 2024-03-12T03:16:35.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi094-79350-1"}]': finished 2024-03-12T03:16:35.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi094-78180-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi094-79269-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.25360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi094-78118-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi094-79269-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi094-78138-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: osdmap e84: 8 total, 8 up, 8 in 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi094-78180-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2007440361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi094-78272-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2469088620' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:35.967 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi094-78118-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:35.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:35.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi094-79269-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:35.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi094-78180-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:35.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi094-78138-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:35.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi094-78272-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:35.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:35.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:35.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi094-78118-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:35.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:35.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-12T03:16:35.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:37.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:37.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-12T03:16:37.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:36 smithi094 ceph-mon[29291]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]': finished 2024-03-12T03:16:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-12T03:16:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:36 smithi094 ceph-mon[40484]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]': finished 2024-03-12T03:16:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-12T03:16:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:36 smithi151 ceph-mon[32923]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]': finished 2024-03-12T03:16:37.491 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.455406+0000 mon.a [INF] from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi094-78180-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi094-78180- cmd: Running main() from gmock_main.cc 2024-03-12T03:16:37.491 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-03-12T03:16:37.491 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: [----------] Global test environment set-up. 2024-03-12T03:16:37.491 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-03-12T03:16:37.491 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-03-12T03:16:37.491 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (9487 ms) 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (3409 ms) 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (7594 ms) 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: [----------] 3 tests from NeoRadosCmd (20490 ms total) 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: [----------] Global test environment tear-down 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: [==========] 3 tests from 1 test suite ran. (20492 ms total) 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: cmd: [ PASSED ] 3 tests. 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: Running main() from gmock_main.cc 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: [----------] Global test environment set-up. 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-03-12T03:16:37.492 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: handle_notify cookie 94794803928512 notify_id 330712481793 notifier_gid 34904 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (10928 ms) 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: Trying... 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: handle_notify cookie 94794815792432 notify_id 347892350977 notifier_gid 25210 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: Waiting for 3.000000000s 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: Timed out. 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: Flushing... 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: Flushed... 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (9440 ms) 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (20368 ms total) 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: [----------] Global test environment tear-down 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (20368 ms total) 2024-03-12T03:16:37.493 INFO:tasks.workunit.client.0.smithi094.stdout: watch_notify: [ PASSED ] 2 tests. 2024-03-12T03:16:37.494 INFO:tasks.workunit.client.0.smithi094.stdout: cls: Running main() from gmock_main.cc 2024-03-12T03:16:37.494 INFO:tasks.workunit.client.0.smithi094.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-03-12T03:16:37.494 INFO:tasks.workunit.client.0.smithi094.stdout: cls: [----------] Global test environment set-up. 2024-03-12T03:16:37.494 INFO:tasks.workunit.client.0.smithi094.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-03-12T03:16:37.494 INFO:tasks.workunit.client.0.smithi094.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-03-12T03:16:37.494 INFO:tasks.workunit.client.0.smithi094.stdout: cls: [ OK ] NeoRadosCls.DNE (12863 ms) 2024-03-12T03:16:37.494 INFO:tasks.workunit.client.0.smithi094.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-03-12T03:16:37.494 INFO:tasks.workunit.client.0.smithi094.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (7586 ms) 2024-03-12T03:16:37.494 INFO:tasks.workunit.client.0.smithi094.stdout: cls: [----------] 2 tests from NeoRadosCls (20449 ms total) 2024-03-12T03:16:37.494 INFO:tasks.workunit.client.0.smithi094.stdout: cls: 2024-03-12T03:16:37.494 INFO:tasks.workunit.client.0.smithi094.stdout: cls: [----------] Global test environment tear-down 2024-03-12T03:16:37.495 INFO:tasks.workunit.client.0.smithi094.stdout: cls: [==========] 2 tests from 1 test suite ran. (20449 ms total) 2024-03-12T03:16:37.495 INFO:tasks.workunit.client.0.smithi094.stdout: cls: [ PASSED ] 2 tests. 2024-03-12T03:16:37.497 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_1_[79105]: starting. 2024-03-12T03:16:37.497 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_1_[79105]: creating pool ceph_test_rados_open_pools_parallel.smithi094-79001 2024-03-12T03:16:37.497 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_1_[79105]: created object 0... 2024-03-12T03:16:37.497 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_1_[79105]: created object 25... 2024-03-12T03:16:37.497 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_1_[79105]: created object 49... 2024-03-12T03:16:37.497 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_1_[79105]: finishing. 2024-03-12T03:16:37.497 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_1_[79105]: shutting down. 2024-03-12T03:16:37.497 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_2_[79106]: starting. 2024-03-12T03:16:37.497 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_2_[79106]: rados_pool_create. 2024-03-12T03:16:37.497 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_2_[79106]: rados_ioctx_create. 2024-03-12T03:16:37.497 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_2_[79106]: shutting down. 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: ******************************* 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_3_[80501]: starting. 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_3_[80501]: creating pool ceph_test_rados_open_pools_parallel.smithi094-79001 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_3_[80501]: created object 0... 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_3_[80501]: created object 25... 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_3_[80501]: created object 49... 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_3_[80501]: finishing. 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_3_[80501]: shutting down. 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: ******************************* 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_4_[80502]: starting. 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_4_[80502]: rados_pool_create. 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_4_[80502]: rados_ioctx_create. 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: process_4_[80502]: shutting down. 2024-03-12T03:16:37.498 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: ******************************* 2024-03-12T03:16:37.499 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: ******************************* 2024-03-12T03:16:37.499 INFO:tasks.workunit.client.0.smithi094.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-12T03:16:37.502 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_1_[79021]: starting. 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_1_[79021]: creating pool ceph_test_rados_delete_pools_parallel.smithi094-78923 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_1_[79021]: created object 0... 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_1_[79021]: created object 25... 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_1_[79021]: created object 49... 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_1_[79021]: finishing. 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_1_[79021]: shutting down. 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_2_[79022]: starting. 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_2_[79022]: deleting pool ceph_test_rados_delete_pools_parallel.smithi094-78923 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_2_[79022]: shutting down. 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: ******************************* 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_3_[80498]: starting. 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_3_[80498]: creating pool ceph_test_rados_delete_pools_parallel.smithi094-78923 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_3_[80498]: created object 0... 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_3_[80498]: created object 25... 2024-03-12T03:16:37.503 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_3_[80498]: created object 49... 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_3_[80498]: finishing. 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_3_[80498]: shutting down. 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: ******************************* 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_5_[80500]: starting. 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_5_[80500]: listing objects. 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_5_[80500]: listed object 0... 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_5_[80500]: listed object 25... 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_5_[80500]: saw 50 objects 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_5_[80500]: shutting down. 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: ******************************* 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_4_[80499]: starting. 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_4_[80499]: deleting pool ceph_test_rados_delete_pools_parallel.smithi094-78923 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: process_4_[80499]: shutting down. 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: ******************************* 2024-03-12T03:16:37.504 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: ******************************* 2024-03-12T03:16:37.505 INFO:tasks.workunit.client.0.smithi094.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-12T03:16:37.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:16:37 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:16:38.016 INFO:tasks.workunit.client.0.smithi094.stdout:10"}]: dispatch 2024-03-12T03:16:38.016 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.455665+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi094-78138-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:38.016 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.455930+0000 mon.a [INF] from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi094-78272-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:38.016 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.456197+0000 mon.a [INF] from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]: dispatch 2024-03-12T03:16:38.016 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.456408+0000 mon.a [INF] from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi094-79350-1"}]: dispatch 2024-03-12T03:16:38.016 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.456635+0000 mon.a [INF] from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi094-78118-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:38.016 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.457149+0000 mon.a [INF] from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:38.016 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:34.609296+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.016 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:35.413630+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-12T03:16:38.016 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:35.495256+0000 mon.a [INF] from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi094-78644-1"}]': finished 2024-03-12T03:16:38.017 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:35.413837+0000 client.admin [INF] threexx 2024-03-12T03:16:38.017 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:35.495318+0000 mon.a [INF] from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi094-79350-1"}]': finished 2024-03-12T03:16:38.017 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:35.495373+0000 mon.a [INF] from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]': finished 2024-03-12T03:16:38.017 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:35.500997+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-4", "overlaypool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:38.017 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:35.527192+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi094-79350-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:38.017 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:36.706691+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1100155812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi094-78081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.017 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:36.719079+0000 mon.a [INF] from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi094-79269-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi094-79269-2"}]': finished 2024-03-12T03:16:38.017 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:36.719170+0000 mon.a [INF] from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi094-78180-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi094-78180-10"}]': finished 2024-03-12T03:16:38.017 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:36.719224+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi094-78138-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi094-78138-22"}]': finished 2024-03-12T03:16:38.017 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:36.719272+0000 mon.a [INF] from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi094-78272-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi094-78272-10"}]': finished 2024-03-12T03:16:38.017 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:36.719315+0000 mon.a [INF] from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi094-78118-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi094-78118-16"}]': finished 2024-03-12T03:16:38.017 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:36.984307+0000 mon.a [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-12T03:16:38.017 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:36.987587+0000 mon.a [INF] from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi094-79350-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:38.018 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:36.989746+0000 mon.a [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.018 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:36.990316+0000 mon.a [INF] from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-4", "overlaypool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:38.018 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.230197+0000 mon.a [INF] from='client.35333 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi094-78081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.018 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.475869+0000 mon.a [INF] from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi094-79350-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:38.018 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.475951+0000 mon.a [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:38.018 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.476023+0000 mon.a [INF] from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-4", "overlaypool": "test-rados-api-smithi094-78452-4-cache"}]': finished 2024-03-12T03:16:38.018 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.476097+0000 mon.a [INF] from='client.35333 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi094-78081-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:38.018 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.480720+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi094-79350-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:38.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: pgmap v84: 852 pgs: 144 unknown, 192 creating+peering, 516 active+clean; 145 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 308 op/s 2024-03-12T03:16:38.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: threexx 2024-03-12T03:16:38.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi094-79350-1"}]': finished 2024-03-12T03:16:38.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]': finished 2024-03-12T03:16:38.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-4", "overlaypool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:38.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi094-79350-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:38.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:38.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: osdmap e85: 8 total, 8 up, 8 in 2024-03-12T03:16:38.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1100155812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi094-78081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi094-79269-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi094-79269-2"}]': finished 2024-03-12T03:16:38.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi094-78180-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi094-78180-10"}]': finished 2024-03-12T03:16:38.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi094-78138-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi094-78138-22"}]': finished 2024-03-12T03:16:38.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi094-78272-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi094-78272-10"}]': finished 2024-03-12T03:16:38.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi094-78118-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi094-78118-16"}]': finished 2024-03-12T03:16:38.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: osdmap e86: 8 total, 8 up, 8 in 2024-03-12T03:16:38.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-12T03:16:38.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi094-79350-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:38.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: pgmap v84: 852 pgs: 144 unknown, 192 creating+peering, 516 active+clean; 145 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 308 op/s 2024-03-12T03:16:38.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: threexx 2024-03-12T03:16:38.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi094-79350-1"}]': finished 2024-03-12T03:16:38.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]': finished 2024-03-12T03:16:38.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-4", "overlaypool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:38.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi094-79350-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:38.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:38.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: osdmap e85: 8 total, 8 up, 8 in 2024-03-12T03:16:38.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1100155812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi094-78081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi094-79269-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi094-79269-2"}]': finished 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi094-78180-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi094-78180-10"}]': finished 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi094-78138-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi094-78138-22"}]': finished 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi094-78272-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi094-78272-10"}]': finished 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi094-78118-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi094-78118-16"}]': finished 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: osdmap e86: 8 total, 8 up, 8 in 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi094-79350-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-4", "overlaypool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.35333 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi094-78081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi094-79350-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:38.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-4", "overlaypool": "test-rados-api-smithi094-78452-4-cache"}]': finished 2024-03-12T03:16:38.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.35333 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi094-78081-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:38.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: osdmap e87: 8 total, 8 up, 8 in 2024-03-12T03:16:38.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi094-79350-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:38.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-4-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:16:38.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4163038171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2"}]: dispatch 2024-03-12T03:16:38.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/906629732' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi094-79350-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:38.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:38.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:38.465 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-4", "overlaypool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:38.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.35333 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi094-78081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:38.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi094-79350-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:38.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:38.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-4", "overlaypool": "test-rados-api-smithi094-78452-4-cache"}]': finished 2024-03-12T03:16:38.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.35333 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi094-78081-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:38.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: osdmap e87: 8 total, 8 up, 8 in 2024-03-12T03:16:38.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi094-79350-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:38.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-4-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:16:38.466 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4163038171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.466 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2"}]: dispatch 2024-03-12T03:16:38.466 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.466 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/906629732' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.466 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi094-79350-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:38.466 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.466 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:38.466 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.466 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:38.466 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:38 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: pgmap v84: 852 pgs: 144 unknown, 192 creating+peering, 516 active+clean; 145 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 308 op/s 2024-03-12T03:16:38.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: threexx 2024-03-12T03:16:38.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi094-79350-1"}]': finished 2024-03-12T03:16:38.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]': finished 2024-03-12T03:16:38.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-4", "overlaypool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:38.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi094-79350-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:38.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:38.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: osdmap e85: 8 total, 8 up, 8 in 2024-03-12T03:16:38.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1100155812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi094-78081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi094-79269-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi094-79269-2"}]': finished 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi094-78180-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi094-78180-10"}]': finished 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi094-78138-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi094-78138-22"}]': finished 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi094-78272-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi094-78272-10"}]': finished 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi094-78118-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi094-78118-16"}]': finished 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: osdmap e86: 8 total, 8 up, 8 in 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi094-79350-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-4", "overlaypool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.35333 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi094-78081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi094-79350-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:38.468 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-4", "overlaypool": "test-rados-api-smithi094-78452-4-cache"}]': finished 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.35333 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi094-78081-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: osdmap e87: 8 total, 8 up, 8 in 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi094-79350-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-4-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4163038171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2"}]: dispatch 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/906629732' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi094-79350-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:38.469 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:38 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:38.637 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: Running main() from gmock_main.cc 2024-03-12T03:16:38.637 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-12T03:16:38.637 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-12T03:16:38.637 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: seed 78138 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (1547 ms) 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (8 ms) 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (4 ms) 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (1191 ms) 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2 ms) 2024-03-12T03:16:38.638 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-12T03:16:38.639 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (3 ms) 2024-03-12T03:16:38.639 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-12T03:16:38.639 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2 ms) 2024-03-12T03:16:38.639 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-12T03:16:38.639 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (8 ms) 2024-03-12T03:16:38.639 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-12T03:16:38.639 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (13 ms) 2024-03-12T03:16:38.639 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-12T03:16:38.639 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (8 ms) 2024-03-12T03:16:38.639 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-12T03:16:38.639 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (9 ms) 2024-03-12T03:16:38.639 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-12T03:16:38.639 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (13 ms) 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (8 ms) 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (5 ms) 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (12 ms) 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (8 ms) 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (4 ms) 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (4 ms) 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-12T03:16:38.640 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (5 ms) 2024-03-12T03:16:38.641 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (2854 ms total) 2024-03-12T03:16:38.641 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: 2024-03-12T03:16:38.641 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-12T03:16:38.641 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-12T03:16:38.641 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1881 ms) 2024-03-12T03:16:38.641 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-12T03:16:38.641 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (16 ms) 2024-03-12T03:16:38.641 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-12T03:16:38.641 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (4 ms) 2024-03-12T03:16:38.641 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-12T03:16:38.641 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (3 ms) 2024-03-12T03:16:38.641 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-12T03:16:38.641 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (2 ms) 2024-03-12T03:16:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: pgmap v86: 948 pgs: 240 unknown, 192 creating+peering, 516 active+clean; 145 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 34 MiB/s wr, 285 op/s 2024-03-12T03:16:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.35237 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-12T03:16:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-12T03:16:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: fourxx 2024-03-12T03:16:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-12T03:16:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-4-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:16:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.25396 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:39.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:16:39.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:39.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]': finished 2024-03-12T03:16:39.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:39.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-4-cache", "mode": "writeback"}]': finished 2024-03-12T03:16:39.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.25396 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:39.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:39.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-12T03:16:39.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:16:39.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:39.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: pgmap v86: 948 pgs: 240 unknown, 192 creating+peering, 516 active+clean; 145 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 34 MiB/s wr, 285 op/s 2024-03-12T03:16:39.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:39.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.35237 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-12T03:16:39.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-12T03:16:39.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: fourxx 2024-03-12T03:16:39.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-12T03:16:39.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-4-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:16:39.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.25396 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:39.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:39.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:39.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:16:39.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:39.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]': finished 2024-03-12T03:16:39.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:39.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-4-cache", "mode": "writeback"}]': finished 2024-03-12T03:16:39.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.25396 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:39.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:39.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-12T03:16:39.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:16:39.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:39.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78644-6", "pg_num": 4}]: dispatch 2024-03-12T03:16:39.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-4"}]: dispatch 2024-03-12T03:16:39.464 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: osdmap e88: 8 total, 8 up, 8 in 2024-03-12T03:16:39.464 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:39.464 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:39.464 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:39.464 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[29291]: from='client.25339 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:16:39.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78644-6", "pg_num": 4}]: dispatch 2024-03-12T03:16:39.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-4"}]: dispatch 2024-03-12T03:16:39.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: osdmap e88: 8 total, 8 up, 8 in 2024-03-12T03:16:39.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:39.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:39.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:39.464 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:39 smithi094 ceph-mon[40484]: from='client.25339 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:16:39.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: pgmap v86: 948 pgs: 240 unknown, 192 creating+peering, 516 active+clean; 145 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 34 MiB/s wr, 285 op/s 2024-03-12T03:16:39.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:39.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.35237 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-12T03:16:39.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3838120283' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-12T03:16:39.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: fourxx 2024-03-12T03:16:39.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.35237 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-12T03:16:39.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-4-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:16:39.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.25396 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:39.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:39.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:39.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:16:39.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:39.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]': finished 2024-03-12T03:16:39.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:39.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-4-cache", "mode": "writeback"}]': finished 2024-03-12T03:16:39.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.25396 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:39.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:39.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-12T03:16:39.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:16:39.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:39.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78644-6", "pg_num": 4}]: dispatch 2024-03-12T03:16:39.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-4"}]: dispatch 2024-03-12T03:16:39.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: osdmap e88: 8 total, 8 up, 8 in 2024-03-12T03:16:39.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:39.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:39.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:39.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:39 smithi151 ceph-mon[32923]: from='client.25339 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:16:39.527 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN api_stat_pp: Running main() from gmock_main.cc 2024-03-12T03:16:39.527 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-12T03:16:39.527 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-12T03:16:39.527 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-12T03:16:39.527 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: seed 78582 2024-03-12T03:16:39.527 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-12T03:16:39.527 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (267 ms) 2024-03-12T03:16:39.527 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-12T03:16:39.527 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (37 ms) 2024-03-12T03:16:39.527 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-12T03:16:39.527 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (3 ms) 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (15 ms) 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (322 ms total) 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1118 ms) 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (2 ms) 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-12T03:16:39.528 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (6 ms) 2024-03-12T03:16:39.529 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1126 ms total) 2024-03-12T03:16:39.529 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: 2024-03-12T03:16:39.529 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-12T03:16:39.529 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (22679 ms total) 2024-03-12T03:16:39.529 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-12T03:16:40.529 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: Running main() from gmock_main.cc 2024-03-12T03:16:40.530 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-12T03:16:40.530 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [----------] Global test environment set-up. 2024-03-12T03:16:40.530 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-12T03:16:40.530 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-12T03:16:40.530 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ OK ] LibRadosStat.Stat (227 ms) 2024-03-12T03:16:40.530 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-12T03:16:40.530 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (9 ms) 2024-03-12T03:16:40.530 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-12T03:16:40.530 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ OK ] LibRadosStat.StatNS (72 ms) 2024-03-12T03:16:40.530 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-12T03:16:40.530 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-03-12T03:16:40.530 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-12T03:16:40.530 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (6 ms) 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [----------] 5 tests from LibRadosStat (315 ms total) 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (3144 ms) 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (57 ms) 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1 ms) 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (3202 ms total) 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: 2024-03-12T03:16:40.531 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [----------] Global test environment tear-down 2024-03-12T03:16:40.532 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (23723 ms total) 2024-03-12T03:16:40.532 INFO:tasks.workunit.client.0.smithi094.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-12T03:16:40.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: pgmap v90: 740 pgs: 64 unknown, 7 creating+activating, 116 creating+peering, 553 active+clean; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-12T03:16:40.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:40.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.35237 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-12T03:16:40.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78644-6", "pg_num": 4}]: dispatch 2024-03-12T03:16:40.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:40.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi094-79350-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi094-79350-2"}]': finished 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]': finished 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.25339 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78644-6", "pg_num": 4}]': finished 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]': finished 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: osdmap e89: 8 total, 8 up, 8 in 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:40.533 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:40.534 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:40.534 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: pgmap v90: 740 pgs: 64 unknown, 7 creating+activating, 116 creating+peering, 553 active+clean; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-12T03:16:40.534 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:40.534 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.35237 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-12T03:16:40.534 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78644-6", "pg_num": 4}]: dispatch 2024-03-12T03:16:40.534 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:40.534 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:40.534 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:40.534 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi094-79350-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi094-79350-2"}]': finished 2024-03-12T03:16:40.534 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]': finished 2024-03-12T03:16:40.535 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.25339 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-12T03:16:40.535 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78644-6", "pg_num": 4}]': finished 2024-03-12T03:16:40.535 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]': finished 2024-03-12T03:16:40.535 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:40.535 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:40.535 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-12T03:16:40.535 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:40.535 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: osdmap e89: 8 total, 8 up, 8 in 2024-03-12T03:16:40.535 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:40.535 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:40.535 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:40.535 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: pgmap v90: 740 pgs: 64 unknown, 7 creating+activating, 116 creating+peering, 553 active+clean; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.35237 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78644-6", "pg_num": 4}]: dispatch 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi094-79350-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi094-79350-2"}]': finished 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi094-78582-7"}]': finished 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.25339 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78644-6", "pg_num": 4}]': finished 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi094-78472-7"}]': finished 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1019512050' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-12T03:16:40.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:40.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: osdmap e89: 8 total, 8 up, 8 in 2024-03-12T03:16:40.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:40.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]: dispatch 2024-03-12T03:16:40.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:40.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:40.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:16:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:16:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:16:41.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.483832+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-4-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:16:41.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.485455+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/4163038171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.485842+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2"}]: dispatch 2024-03-12T03:16:41.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.486604+0000 mon.b [INF] from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi094-78284-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.488706+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/906629732' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.491658+0000 mon.a [INF] from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi094-79350-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:41.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.492189+0000 mon.a [INF] from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.026 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.495202+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1928228574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:41.026 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.496104+0000 mon.c [INF] from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.026 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.500182+0000 mon.a [INF] from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi094-78582-7"}]: dispatch 2024-03-12T03:16:41.026 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: got: 2024-03-12T03:16:37.501423+0000 mon.a [INF] from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.026 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (11130 ms) 2024-03-12T03:16:41.026 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (24156 ms total) 2024-03-12T03:16:41.026 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: 2024-03-12T03:16:41.026 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-12T03:16:41.026 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (24156 ms total) 2024-03-12T03:16:41.026 INFO:tasks.workunit.client.0.smithi094.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-12T03:16:41.526 INFO:tasks.workunit.client.0.smithi094.stdout: ec_io: Running main() from gmock_main.cc 2024-03-12T03:16:41.526 INFO:tasks.workunit.client.0.smithi094.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-03-12T03:16:41.526 INFO:tasks.workunit.client.0.smithi094.stdout: ec_io: [----------] Global test environment set-up. 2024-03-12T03:16:41.526 INFO:tasks.workunit.client.0.smithi094.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-03-12T03:16:41.526 INFO:tasks.workunit.client.0.smithi094.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-03-12T03:16:41.526 INFO:tasks.workunit.client.0.smithi094.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (16349 ms) 2024-03-12T03:16:41.526 INFO:tasks.workunit.client.0.smithi094.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-03-12T03:16:41.526 INFO:tasks.workunit.client.0.smithi094.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (8099 ms) 2024-03-12T03:16:41.526 INFO:tasks.workunit.client.0.smithi094.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (24448 ms total) 2024-03-12T03:16:41.526 INFO:tasks.workunit.client.0.smithi094.stdout: ec_io: 2024-03-12T03:16:41.526 INFO:tasks.workunit.client.0.smithi094.stdout: ec_io: [----------] Global test environment tear-down 2024-03-12T03:16:41.526 INFO:tasks.workunit.client.0.smithi094.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (24448 ms total) 2024-03-12T03:16:41.527 INFO:tasks.workunit.client.0.smithi094.stdout: ec_io: [ PASSED ] 2 tests. 2024-03-12T03:16:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-4"}]: dispatch 2024-03-12T03:16:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.25339 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]': finished 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-4"}]': finished 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi094-79269-2"}]': finished 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.25339 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: osdmap e90: 8 total, 8 up, 8 in 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi094-78601-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "overlaypool": "test-rados-api-smithi094-78644-6"}]: dispatch 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:41.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:41.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:41.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1"}]: dispatch 2024-03-12T03:16:41.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:41.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:41.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.25339 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1"}]: dispatch 2024-03-12T03:16:41.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2362949338' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi094-78081-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi094-78601-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:41.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "overlaypool": "test-rados-api-smithi094-78644-6"}]: dispatch 2024-03-12T03:16:41.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi094-78081-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/312685365' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[29291]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-4"}]: dispatch 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.25339 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]': finished 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-4"}]': finished 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi094-79269-2"}]': finished 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.25339 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: osdmap e90: 8 total, 8 up, 8 in 2024-03-12T03:16:41.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:41.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi094-78601-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:41.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "overlaypool": "test-rados-api-smithi094-78644-6"}]: dispatch 2024-03-12T03:16:41.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:41.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:41.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:41.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:41.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:41.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1"}]: dispatch 2024-03-12T03:16:41.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:41.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:41.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.25339 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1"}]: dispatch 2024-03-12T03:16:41.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2362949338' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi094-78081-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi094-78601-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:41.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "overlaypool": "test-rados-api-smithi094-78644-6"}]: dispatch 2024-03-12T03:16:41.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi094-78081-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/312685365' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:41 smithi094 ceph-mon[40484]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:41.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:41.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-4"}]: dispatch 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.25339 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi094-78472-7"}]': finished 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-4"}]': finished 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi094-79269-2"}]': finished 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.25339 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: osdmap e90: 8 total, 8 up, 8 in 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/391288672' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi094-78601-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "overlaypool": "test-rados-api-smithi094-78644-6"}]: dispatch 2024-03-12T03:16:41.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2391492140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi094-79269-2"}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2256068162' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1"}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.25339 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1"}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2362949338' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi094-78081-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi094-78601-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "overlaypool": "test-rados-api-smithi094-78644-6"}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi094-78081-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/312685365' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:41.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:41 smithi151 ceph-mon[32923]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:42.546 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: Running main() from gmock_main.cc 2024-03-12T03:16:42.546 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-12T03:16:42.546 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [----------] Global test environment set-up. 2024-03-12T03:16:42.546 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (304 ms) 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: no timeout :/ 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: no timeout :/ 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: no timeout :/ 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: no timeout :/ 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: no timeout :/ 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (49 ms) 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (9 ms) 2024-03-12T03:16:42.547 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.Checksum (4 ms) 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (6 ms) 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (4 ms) 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (6 ms) 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (5 ms) 2024-03-12T03:16:42.548 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (6 ms) 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.RmXattr (11 ms) 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIo.XattrIter (6 ms) 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [----------] 14 tests from LibRadosIo (416 ms total) 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (883 ms) 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (21 ms) 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-12T03:16:42.549 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (985 ms) 2024-03-12T03:16:42.550 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-12T03:16:42.550 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-03-12T03:16:42.550 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-12T03:16:42.550 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (7 ms) 2024-03-12T03:16:42.550 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-12T03:16:42.550 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (3 ms) 2024-03-12T03:16:42.550 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-12T03:16:42.550 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-03-12T03:16:42.550 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-12T03:16:42.550 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (4 ms) 2024-03-12T03:16:42.550 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-12T03:16:42.550 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (9 ms) 2024-03-12T03:16:42.550 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (5 ms) 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1926 ms total) 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [----------] Global test environment tear-down 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [==========] 24 tests from 2 test suites ran. (25857 ms total) 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: api_io: [ PASSED ] 24 tests. 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (4 ms) 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5 ms) 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (2 ms) 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-12T03:16:42.551 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (7 ms) 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (4 ms) 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (2 ms) 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (8 ms) 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (4 ms) 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-12T03:16:42.552 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (2 ms) 2024-03-12T03:16:42.553 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-12T03:16:42.553 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (4 ms) 2024-03-12T03:16:42.553 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1955 ms total) 2024-03-12T03:16:42.553 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: 2024-03-12T03:16:42.553 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-12T03:16:42.553 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (25850 ms total) 2024-03-12T03:16:42.553 INFO:tasks.workunit.client.0.smithi094.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-12T03:16:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: pgmap v93: 800 pgs: 204 unknown, 3 creating+activating, 43 creating+peering, 550 active+clean; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 2.1 KiB/s rd, 1 op/s 2024-03-12T03:16:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:16:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]': finished 2024-03-12T03:16:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi094-79269-2"}]': finished 2024-03-12T03:16:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]': finished 2024-03-12T03:16:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]': finished 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.25339 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1"}]': finished 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "overlaypool": "test-rados-api-smithi094-78644-6"}]': finished 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi094-78081-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78644-6", "mode": "writeback"}]: dispatch 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: osdmap e91: 8 total, 8 up, 8 in 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78644-6", "mode": "writeback"}]: dispatch 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:42.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/623959237' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:42.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:42.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:42.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[29291]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:42.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: pgmap v93: 800 pgs: 204 unknown, 3 creating+activating, 43 creating+peering, 550 active+clean; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 2.1 KiB/s rd, 1 op/s 2024-03-12T03:16:42.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:42.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:16:42.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]': finished 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi094-79269-2"}]': finished 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]': finished 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]': finished 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.25339 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1"}]': finished 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "overlaypool": "test-rados-api-smithi094-78644-6"}]': finished 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi094-78081-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78644-6", "mode": "writeback"}]: dispatch 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: osdmap e91: 8 total, 8 up, 8 in 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:42.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78644-6", "mode": "writeback"}]: dispatch 2024-03-12T03:16:42.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:42.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:42.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:42.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/623959237' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:42.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:42.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:42.964 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:42 smithi094 ceph-mon[40484]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: pgmap v93: 800 pgs: 204 unknown, 3 creating+activating, 43 creating+peering, 550 active+clean; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 2.1 KiB/s rd, 1 op/s 2024-03-12T03:16:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:16:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-4", "tierpool": "test-rados-api-smithi094-78452-4-cache"}]': finished 2024-03-12T03:16:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi094-79269-2"}]': finished 2024-03-12T03:16:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi094-78138-22"}]': finished 2024-03-12T03:16:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi094-78118-16"}]': finished 2024-03-12T03:16:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.25339 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi094-78284-1","app":"app1","key":"key1"}]': finished 2024-03-12T03:16:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "overlaypool": "test-rados-api-smithi094-78644-6"}]': finished 2024-03-12T03:16:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi094-78081-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78644-6", "mode": "writeback"}]: dispatch 2024-03-12T03:16:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2187101095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:42.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2691820375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:42.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: osdmap e91: 8 total, 8 up, 8 in 2024-03-12T03:16:42.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]: dispatch 2024-03-12T03:16:42.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:42.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78644-6", "mode": "writeback"}]: dispatch 2024-03-12T03:16:42.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:42.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]: dispatch 2024-03-12T03:16:42.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:42.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/623959237' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:42.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:42.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:42.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:42 smithi151 ceph-mon[32923]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:43.994 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-12T03:16:43.994 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-12T03:16:43.994 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-12T03:16:43.994 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-12T03:16:43.994 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: seed 78272 2024-03-12T03:16:43.994 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-12T03:16:43.994 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (257 ms) 2024-03-12T03:16:43.994 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (17 ms) 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1088 ms) 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1087 ms) 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (5 ms) 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (6 ms) 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (4 ms) 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-03-12T03:16:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2467 ms total) 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1885 ms) 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (5 ms) 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1048 ms) 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1031 ms) 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (3 ms) 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (5 ms) 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-12T03:16:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (5 ms) 2024-03-12T03:16:43.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-12T03:16:43.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-03-12T03:16:43.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3985 ms total) 2024-03-12T03:16:43.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: 2024-03-12T03:16:43.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-12T03:16:43.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (27256 ms total) 2024-03-12T03:16:43.997 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-12T03:16:44.021 INFO:tasks.workunit.client.0.smithi094.stdout: pool: Running main() from gmock_main.cc 2024-03-12T03:16:44.021 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-03-12T03:16:44.021 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [----------] Global test environment set-up. 2024-03-12T03:16:44.021 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-03-12T03:16:44.021 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-03-12T03:16:44.021 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [ OK ] NeoRadosPools.PoolList (10009 ms) 2024-03-12T03:16:44.021 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-03-12T03:16:44.021 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (3822 ms) 2024-03-12T03:16:44.021 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-03-12T03:16:44.022 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (3511 ms) 2024-03-12T03:16:44.022 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-03-12T03:16:44.022 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (5069 ms) 2024-03-12T03:16:44.022 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-03-12T03:16:44.022 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2328 ms) 2024-03-12T03:16:44.022 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-03-12T03:16:44.022 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2165 ms) 2024-03-12T03:16:44.022 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [----------] 6 tests from NeoRadosPools (26904 ms total) 2024-03-12T03:16:44.022 INFO:tasks.workunit.client.0.smithi094.stdout: pool: 2024-03-12T03:16:44.022 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [----------] Global test environment tear-down 2024-03-12T03:16:44.022 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [==========] 6 tests from 1 test suite ran. (26904 ms total) 2024-03-12T03:16:44.022 INFO:tasks.workunit.client.0.smithi094.stdout: pool: [ PASSED ] 6 tests. 2024-03-12T03:16:44.023 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: Running main() from gmock_main.cc 2024-03-12T03:16:44.023 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-12T03:16:44.023 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [----------] Global test environment set-up. 2024-03-12T03:16:44.023 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-12T03:16:44.023 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-12T03:16:44.023 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (275 ms) 2024-03-12T03:16:44.023 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-12T03:16:44.023 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLock.LockShared (28 ms) 2024-03-12T03:16:44.023 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-12T03:16:44.023 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1065 ms) 2024-03-12T03:16:44.023 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-12T03:16:44.023 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1085 ms) 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (4 ms) 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLock.Unlock (5 ms) 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (4 ms) 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (4 ms) 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [----------] 8 tests from LibRadosLock (2470 ms total) 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1885 ms) 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-12T03:16:44.024 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (7 ms) 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1043 ms) 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1032 ms) 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (4 ms) 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4 ms) 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (6 ms) 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (2 ms) 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3983 ms total) 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: 2024-03-12T03:16:44.025 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [----------] Global test environment tear-down 2024-03-12T03:16:44.026 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (27320 ms total) 2024-03-12T03:16:44.026 INFO:tasks.workunit.client.0.smithi094.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-12T03:16:44.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:44.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: pool 'PoolQuotaPP_smithi094-78099-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-12T03:16:44.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:16:44.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-12T03:16:44.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi094-78601-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]': finished 2024-03-12T03:16:44.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]': finished 2024-03-12T03:16:44.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78644-6", "mode": "writeback"}]': finished 2024-03-12T03:16:44.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]': finished 2024-03-12T03:16:44.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/623959237' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:44.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]': finished 2024-03-12T03:16:44.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi094-79350-2"}]': finished 2024-03-12T03:16:44.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]': finished 2024-03-12T03:16:44.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: osdmap e92: 8 total, 8 up, 8 in 2024-03-12T03:16:44.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:44.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:44.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:44.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:44.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:44.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:44.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:44.217 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:44.217 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: pool 'PoolQuotaPP_smithi094-78099-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-12T03:16:44.217 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:16:44.217 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-12T03:16:44.217 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi094-78601-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]': finished 2024-03-12T03:16:44.217 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]': finished 2024-03-12T03:16:44.217 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78644-6", "mode": "writeback"}]': finished 2024-03-12T03:16:44.217 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]': finished 2024-03-12T03:16:44.217 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/623959237' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:44.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]': finished 2024-03-12T03:16:44.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi094-79350-2"}]': finished 2024-03-12T03:16:44.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]': finished 2024-03-12T03:16:44.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: osdmap e92: 8 total, 8 up, 8 in 2024-03-12T03:16:44.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:44.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:44.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:44.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:44.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:44.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:44.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:44.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:44.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: pool 'PoolQuotaPP_smithi094-78099-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-12T03:16:44.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:16:44.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-12T03:16:44.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi094-78601-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]': finished 2024-03-12T03:16:44.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi094-78118-16"}]': finished 2024-03-12T03:16:44.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78644-6", "mode": "writeback"}]': finished 2024-03-12T03:16:44.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi094-78138-22"}]': finished 2024-03-12T03:16:44.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/623959237' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:44.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi094-78272-10"}]': finished 2024-03-12T03:16:44.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi094-79350-2"}]': finished 2024-03-12T03:16:44.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi094-78180-10"}]': finished 2024-03-12T03:16:44.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: osdmap e92: 8 total, 8 up, 8 in 2024-03-12T03:16:44.220 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1714972754' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:44.220 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1252160889' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:44.220 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi094-79350-2"}]: dispatch 2024-03-12T03:16:44.220 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1464103396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:44.220 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]: dispatch 2024-03-12T03:16:44.220 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]: dispatch 2024-03-12T03:16:44.220 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[29291]: pgmap v96: 656 pgs: 140 unknown, 32 creating+peering, 484 active+clean; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:16:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[29291]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi094-79350-2"}]': finished 2024-03-12T03:16:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[29291]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]': finished 2024-03-12T03:16:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[29291]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]': finished 2024-03-12T03:16:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2746540424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi094-78081-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2077118054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi094-78434-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2775502452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi094-79350-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[29291]: osdmap e93: 8 total, 8 up, 8 in 2024-03-12T03:16:45.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[29291]: from='client.36317 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi094-78434-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:45.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[29291]: from='client.26452 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi094-78081-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:45.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[40484]: pgmap v96: 656 pgs: 140 unknown, 32 creating+peering, 484 active+clean; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:16:45.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[40484]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi094-79350-2"}]': finished 2024-03-12T03:16:45.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[40484]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]': finished 2024-03-12T03:16:45.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[40484]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]': finished 2024-03-12T03:16:45.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2746540424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi094-78081-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:45.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2077118054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi094-78434-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:45.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2775502452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi094-79350-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:45.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:45.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[40484]: osdmap e93: 8 total, 8 up, 8 in 2024-03-12T03:16:45.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[40484]: from='client.36317 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi094-78434-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:45.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:44 smithi094 ceph-mon[40484]: from='client.26452 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi094-78081-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:44 smithi151 ceph-mon[32923]: pgmap v96: 656 pgs: 140 unknown, 32 creating+peering, 484 active+clean; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:16:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:44 smithi151 ceph-mon[32923]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi094-79350-2"}]': finished 2024-03-12T03:16:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:44 smithi151 ceph-mon[32923]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi094-78180-10"}]': finished 2024-03-12T03:16:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:44 smithi151 ceph-mon[32923]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi094-78272-10"}]': finished 2024-03-12T03:16:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2746540424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi094-78081-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2077118054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi094-78434-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2775502452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi094-79350-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:44 smithi151 ceph-mon[32923]: osdmap e93: 8 total, 8 up, 8 in 2024-03-12T03:16:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:44 smithi151 ceph-mon[32923]: from='client.36317 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi094-78434-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:44 smithi151 ceph-mon[32923]: from='client.26452 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi094-78081-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:45.882 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-12T03:16:45.882 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-12T03:16:45.882 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-12T03:16:45.882 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-12T03:16:45.882 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-12T03:16:45.882 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: notify 2024-03-12T03:16:45.882 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1213 ms) 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (11 ms) 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1224 ms total) 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: notify 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (110 ms) 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: notify 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4107 ms) 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (25 ms) 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-12T03:16:45.883 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (21 ms) 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: handle_notify cookie 94294924314672 notify_id 395136991232 notifier_gid 35348 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (25 ms) 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: handle_notify cookie 94294924335152 notify_id 395136991232 notifier_gid 35348 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (18 ms) 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: handle_notify cookie 94294924410352 notify_id 395136991233 notifier_gid 35348 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (16 ms) 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: handle_notify cookie 94294924417056 notify_id 395136991233 notifier_gid 35348 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (24 ms) 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: handle_notify cookie 94294924417056 notify_id 395136991234 notifier_gid 35348 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (14 ms) 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-12T03:16:45.884 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: handle_notify cookie 94294924410352 notify_id 395136991235 notifier_gid 35348 2024-03-12T03:16:45.885 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (14 ms) 2024-03-12T03:16:45.885 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-12T03:16:45.885 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: trying... 2024-03-12T03:16:45.885 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: handle_notify cookie 94294924410352 notify_id 395136991236 notifier_gid 35348 2024-03-12T03:16:45.885 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: timed out 2024-03-12T03:16:45.885 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: flushing 2024-03-12T03:16:46.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:45 smithi094 ceph-mon[29291]: pgmap v98: 656 pgs: 39 creating+peering, 129 unknown, 488 active+clean; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-03-12T03:16:46.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:45 smithi094 ceph-mon[29291]: from='client.28036 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi094-79350-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:46.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:45 smithi094 ceph-mon[29291]: from='client.36317 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi094-78434-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:46.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:45 smithi094 ceph-mon[29291]: from='client.26452 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi094-78081-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:46.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:45 smithi094 ceph-mon[29291]: osdmap e94: 8 total, 8 up, 8 in 2024-03-12T03:16:46.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3801019174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:46.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:46.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:45 smithi094 ceph-mon[29291]: from='client.37673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:46.369 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:45 smithi094 ceph-mon[29291]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:46.369 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2672466449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:46.369 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:45 smithi094 ceph-mon[29291]: from='client.36338 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:46.369 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:46.369 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:46 smithi094 ceph-mon[40484]: pgmap v98: 656 pgs: 39 creating+peering, 129 unknown, 488 active+clean; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-03-12T03:16:46.369 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:46 smithi094 ceph-mon[40484]: from='client.28036 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi094-79350-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:46.369 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:46 smithi094 ceph-mon[40484]: from='client.36317 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi094-78434-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:46.369 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:46 smithi094 ceph-mon[40484]: from='client.26452 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi094-78081-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:46.369 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:46 smithi094 ceph-mon[40484]: osdmap e94: 8 total, 8 up, 8 in 2024-03-12T03:16:46.369 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3801019174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:46.369 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:46.369 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:46 smithi094 ceph-mon[40484]: from='client.37673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:46.370 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:46 smithi094 ceph-mon[40484]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:46.370 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2672466449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:46.370 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:46 smithi094 ceph-mon[40484]: from='client.36338 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:46.370 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:46.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:45 smithi151 ceph-mon[32923]: pgmap v98: 656 pgs: 39 creating+peering, 129 unknown, 488 active+clean; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-03-12T03:16:46.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:45 smithi151 ceph-mon[32923]: from='client.28036 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi094-79350-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:16:46.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:45 smithi151 ceph-mon[32923]: from='client.36317 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi094-78434-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:46.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:45 smithi151 ceph-mon[32923]: from='client.26452 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi094-78081-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:46.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:45 smithi151 ceph-mon[32923]: osdmap e94: 8 total, 8 up, 8 in 2024-03-12T03:16:46.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3801019174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:46.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:46.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:45 smithi151 ceph-mon[32923]: from='client.37673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:46.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:45 smithi151 ceph-mon[32923]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:46.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2672466449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:46.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:45 smithi151 ceph-mon[32923]: from='client.36338 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:46.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:47.066 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-03-12T03:16:47.067 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-12T03:16:47.067 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-12T03:16:47.067 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-12T03:16:47.067 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-12T03:16:47.067 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify_test_cb 2024-03-12T03:16:47.067 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (315 ms) 2024-03-12T03:16:47.067 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-12T03:16:47.067 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94715804679104 err -107 2024-03-12T03:16:47.067 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-12T03:16:47.067 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (13 ms) 2024-03-12T03:16:47.067 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-12T03:16:47.067 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-12T03:16:47.067 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94715804681232 err -107 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1064 ms) 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify2_test_cb from 24757 notify_id 335007449089 cookie 94715804703616 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (17 ms) 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify2_test_cb from 24757 notify_id 335007449090 cookie 94715804703616 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (6 ms) 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify2_test_cb from 24757 notify_id 335007449088 cookie 94715804707872 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (3 ms) 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify2_test_cb from 24757 notify_id 335007449089 cookie 94715804707872 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify2_test_cb from 24757 notify_id 335007449089 cookie 94715804734832 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (7 ms) 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-12T03:16:47.068 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify2_test_cb from 24757 notify_id 335007449091 cookie 94715804734832 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify2_test_cb from 24757 notify_id 339302416388 cookie 94715804734832 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3004 ms) 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94715804734832 err -107 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify2_test_cb from 24757 notify_id 365072220161 cookie 94715804734832 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5008 ms) 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94715804773136 err -107 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1005 ms) 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10442 ms total) 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-12T03:16:47.069 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: watch_notify_test_cb 2024-03-12T03:16:47.070 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1548 ms) 2024-03-12T03:16:47.070 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1548 ms total) 2024-03-12T03:16:47.070 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: 2024-03-12T03:16:47.070 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-12T03:16:47.070 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (30221 ms total) 2024-03-12T03:16:47.070 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-12T03:16:47.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-12T03:16:47.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-12T03:16:47.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-12T03:16:47.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-12T03:16:47.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5051 ms) 2024-03-12T03:16:47.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-12T03:16:47.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (39 ms) 2024-03-12T03:16:47.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-12T03:16:47.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20045 ms) 2024-03-12T03:16:47.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-12T03:16:47.191 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: attempt 0 of 20 2024-03-12T03:16:47.191 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5134 ms) 2024-03-12T03:16:47.191 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30269 ms total) 2024-03-12T03:16:47.191 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: 2024-03-12T03:16:47.191 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-12T03:16:47.191 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30269 ms total) 2024-03-12T03:16:47.191 INFO:tasks.workunit.client.0.smithi094.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-12T03:16:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[29291]: from='client.28036 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi094-79350-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[29291]: from='client.37673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[29291]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]': finished 2024-03-12T03:16:47.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[29291]: from='client.36338 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:47.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[29291]: osdmap e95: 8 total, 8 up, 8 in 2024-03-12T03:16:47.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2775502452' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi094-79350-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:47.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[29291]: from='client.28036 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi094-79350-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:47.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:47.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:47.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[29291]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:47.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:47.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:47.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[40484]: from='client.28036 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi094-79350-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:47.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[40484]: from='client.37673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:47.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[40484]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]': finished 2024-03-12T03:16:47.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[40484]: from='client.36338 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:47.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[40484]: osdmap e95: 8 total, 8 up, 8 in 2024-03-12T03:16:47.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2775502452' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi094-79350-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:47.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[40484]: from='client.28036 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi094-79350-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:47.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:47.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:47.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[40484]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:47.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:47.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:47 smithi151 ceph-mon[32923]: from='client.28036 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi094-79350-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:16:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:47 smithi151 ceph-mon[32923]: from='client.37673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:47 smithi151 ceph-mon[32923]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi094-78601-12"}]': finished 2024-03-12T03:16:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:47 smithi151 ceph-mon[32923]: from='client.36338 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:47 smithi151 ceph-mon[32923]: osdmap e95: 8 total, 8 up, 8 in 2024-03-12T03:16:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2775502452' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi094-79350-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:47.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:47 smithi151 ceph-mon[32923]: from='client.28036 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi094-79350-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:47.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1133850880' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:47.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:47.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:47 smithi151 ceph-mon[32923]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]: dispatch 2024-03-12T03:16:47.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:47 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:47.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:47.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:16:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:16:48.221 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: client: 1023 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-12T03:16:48.221 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:48.221 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:48.221 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: cluster: 2024-03-12T03:16:48.221 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: id: 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:16:48.221 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: health: HEALTH_WARN 2024-03-12T03:16:48.221 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 29 pool(s) do not have an application enabled 2024-03-12T03:16:48.221 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:48.222 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: services: 2024-03-12T03:16:48.222 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-12T03:16:48.222 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-12T03:16:48.222 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-12T03:16:48.222 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-12T03:16:48.222 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-03-12T03:16:48.222 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-12T03:16:48.222 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:48.222 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: data: 2024-03-12T03:16:48.222 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: pools: 36 pools, 1092 pgs 2024-03-12T03:16:48.222 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-12T03:16:48.223 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: usage: 227 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:16:48.223 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: pgs: 35.165% pgs unknown 2024-03-12T03:16:48.223 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 52.747% pgs not active 2024-03-12T03:16:48.223 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 576 creating+peering 2024-03-12T03:16:48.223 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 384 unknown 2024-03-12T03:16:48.223 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 132 active+clean 2024-03-12T03:16:48.223 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:48.223 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: io: 2024-03-12T03:16:48.223 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: client: 729 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-12T03:16:48.223 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:48.223 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:48.224 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [ OK ] LibRadosService.StatusFormat (6188 ms) 2024-03-12T03:16:48.224 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-12T03:16:48.224 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [ OK ] LibRadosService.Status (20022 ms) 2024-03-12T03:16:48.224 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [----------] 4 tests from LibRadosService (31284 ms total) 2024-03-12T03:16:48.224 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: 2024-03-12T03:16:48.224 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [----------] Global test environment tear-down 2024-03-12T03:16:48.224 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31284 ms total) 2024-03-12T03:16:48.224 INFO:tasks.workunit.client.0.smithi094.stdout: api_service: [ PASSED ] 4 tests. 2024-03-12T03:16:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[29291]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[29291]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]': finished 2024-03-12T03:16:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1542147466' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-12T03:16:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[29291]: osdmap e96: 8 total, 8 up, 8 in 2024-03-12T03:16:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1246424257' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi094-78081-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:48.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[29291]: from='client.28036 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi094-79350-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi094-79350-3"}]': finished 2024-03-12T03:16:48.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[29291]: osdmap e97: 8 total, 8 up, 8 in 2024-03-12T03:16:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[40484]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[40484]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]': finished 2024-03-12T03:16:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1542147466' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-12T03:16:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[40484]: osdmap e96: 8 total, 8 up, 8 in 2024-03-12T03:16:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1246424257' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi094-78081-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[40484]: from='client.28036 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi094-79350-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi094-79350-3"}]': finished 2024-03-12T03:16:48.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:48 smithi094 ceph-mon[40484]: osdmap e97: 8 total, 8 up, 8 in 2024-03-12T03:16:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:48 smithi151 ceph-mon[32923]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:48 smithi151 ceph-mon[32923]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi094-78601-12"}]': finished 2024-03-12T03:16:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:48 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1542147466' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-12T03:16:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:48.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:48 smithi151 ceph-mon[32923]: osdmap e96: 8 total, 8 up, 8 in 2024-03-12T03:16:48.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1246424257' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi094-78081-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:48.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:48 smithi151 ceph-mon[32923]: from='client.28036 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi094-79350-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi094-79350-3"}]': finished 2024-03-12T03:16:48.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:48 smithi151 ceph-mon[32923]: osdmap e97: 8 total, 8 up, 8 in 2024-03-12T03:16:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: pgmap v101: 680 pgs: 224 unknown, 456 active+clean; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.9 KiB/s wr, 2 op/s 2024-03-12T03:16:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-12T03:16:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: from='client.30502 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi094-78081-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-12T03:16:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: from='client.30502 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi094-78081-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: osdmap e98: 8 total, 8 up, 8 in 2024-03-12T03:16:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4142495608' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:16:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[40484]: from='client.31873 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: pgmap v101: 680 pgs: 224 unknown, 456 active+clean; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.9 KiB/s wr, 2 op/s 2024-03-12T03:16:50.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-12T03:16:50.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:50.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: from='client.30502 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi094-78081-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:50.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-12T03:16:50.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: from='client.30502 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi094-78081-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:50.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: osdmap e98: 8 total, 8 up, 8 in 2024-03-12T03:16:50.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4142495608' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:50.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:16:50.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:49 smithi094 ceph-mon[29291]: from='client.31873 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: pgmap v101: 680 pgs: 224 unknown, 456 active+clean; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.9 KiB/s wr, 2 op/s 2024-03-12T03:16:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-12T03:16:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: from='client.30502 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi094-78081-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-12T03:16:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: from='client.30502 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi094-78081-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: osdmap e98: 8 total, 8 up, 8 in 2024-03-12T03:16:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4142495608' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:16:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:49 smithi151 ceph-mon[32923]: from='client.31873 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:50.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:16:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:16:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:16:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[29291]: pgmap v104: 688 pgs: 40 unknown, 64 creating+peering, 584 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-12T03:16:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[29291]: from='client.31873 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[29291]: osdmap e99: 8 total, 8 up, 8 in 2024-03-12T03:16:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1570632005' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi094-78440-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[29291]: from='client.33550 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi094-78440-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:51.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:16:51.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:51.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:51.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[40484]: pgmap v104: 688 pgs: 40 unknown, 64 creating+peering, 584 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-12T03:16:51.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:51.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:51.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[40484]: from='client.31873 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:51.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[40484]: osdmap e99: 8 total, 8 up, 8 in 2024-03-12T03:16:51.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1570632005' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi094-78440-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:51.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[40484]: from='client.33550 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi094-78440-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:51.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:51.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:16:51.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:51.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:51 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:51.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:51 smithi151 ceph-mon[32923]: pgmap v104: 688 pgs: 40 unknown, 64 creating+peering, 584 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-12T03:16:51.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:51 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:51 smithi151 ceph-mon[32923]: from='client.31873 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:51 smithi151 ceph-mon[32923]: osdmap e99: 8 total, 8 up, 8 in 2024-03-12T03:16:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1570632005' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi094-78440-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:51 smithi151 ceph-mon[32923]: from='client.33550 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi094-78440-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:51 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:16:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:51 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:52.909 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:52 smithi151 ceph-mon[32923]: pgmap v107: 656 pgs: 200 unknown, 32 creating+peering, 424 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3 op/s 2024-03-12T03:16:52.909 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:52 smithi151 ceph-mon[32923]: from='client.33550 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi094-78440-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:52.909 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:16:52.909 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:52 smithi151 ceph-mon[32923]: osdmap e100: 8 total, 8 up, 8 in 2024-03-12T03:16:52.910 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-7"}]: dispatch 2024-03-12T03:16:52.910 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-7"}]: dispatch 2024-03-12T03:16:52.910 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4032487180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:52.910 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/762010067' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:52.910 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:52 smithi151 ceph-mon[32923]: from='client.34741 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:52.910 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[40484]: pgmap v107: 656 pgs: 200 unknown, 32 creating+peering, 424 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3 op/s 2024-03-12T03:16:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[40484]: from='client.33550 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi094-78440-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:16:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[40484]: osdmap e100: 8 total, 8 up, 8 in 2024-03-12T03:16:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-7"}]: dispatch 2024-03-12T03:16:52.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-7"}]: dispatch 2024-03-12T03:16:52.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4032487180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:52.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/762010067' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:52.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[40484]: from='client.34741 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:52.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:52.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[29291]: pgmap v107: 656 pgs: 200 unknown, 32 creating+peering, 424 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3 op/s 2024-03-12T03:16:52.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[29291]: from='client.33550 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi094-78440-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:52.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:16:52.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[29291]: osdmap e100: 8 total, 8 up, 8 in 2024-03-12T03:16:52.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-7"}]: dispatch 2024-03-12T03:16:52.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-7"}]: dispatch 2024-03-12T03:16:52.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4032487180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:52.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/762010067' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:52.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[29291]: from='client.34741 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:52.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:54.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-7"}]': finished 2024-03-12T03:16:54.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4032487180' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:54.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.34741 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: osdmap e101: 8 total, 8 up, 8 in 2024-03-12T03:16:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/615934539' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3638688137' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi094-78081-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:16:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.44744 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi094-78081-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.35965 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:16:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.44744 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi094-78081-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.35965 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:16:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: osdmap e102: 8 total, 8 up, 8 in 2024-03-12T03:16:54.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7"}]: dispatch 2024-03-12T03:16:54.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2775502452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:54.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7"}]: dispatch 2024-03-12T03:16:54.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.28036 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:54.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:54.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-7"}]': finished 2024-03-12T03:16:54.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4032487180' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:54.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.34741 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:54.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: osdmap e101: 8 total, 8 up, 8 in 2024-03-12T03:16:54.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/615934539' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:54.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3638688137' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi094-78081-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.44744 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi094-78081-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.35965 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.44744 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi094-78081-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.35965 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: osdmap e102: 8 total, 8 up, 8 in 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7"}]: dispatch 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2775502452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7"}]: dispatch 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.28036 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:54.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-7"}]': finished 2024-03-12T03:16:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4032487180' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.34741 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: osdmap e101: 8 total, 8 up, 8 in 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/615934539' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3638688137' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi094-78081-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.44744 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi094-78081-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.35965 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.44744 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi094-78081-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.35965 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78764-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: osdmap e102: 8 total, 8 up, 8 in 2024-03-12T03:16:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7"}]: dispatch 2024-03-12T03:16:54.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2775502452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:54.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7"}]: dispatch 2024-03-12T03:16:54.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.28036 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:54.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:55.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:55 smithi151 ceph-mon[32923]: pgmap v110: 720 pgs: 296 unknown, 424 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:16:55.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:55 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7"}]': finished 2024-03-12T03:16:55.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:55 smithi151 ceph-mon[32923]: from='client.28036 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi094-79350-3"}]': finished 2024-03-12T03:16:55.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:55 smithi151 ceph-mon[32923]: osdmap e103: 8 total, 8 up, 8 in 2024-03-12T03:16:55.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2775502452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:55.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2734822432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:55.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:55 smithi151 ceph-mon[32923]: from='client.46229 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:55.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:55 smithi151 ceph-mon[32923]: from='client.28036 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:55.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:55.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4"}]: dispatch 2024-03-12T03:16:55.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:55 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4"}]: dispatch 2024-03-12T03:16:55.282 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: Running main() from gmock_main.cc 2024-03-12T03:16:55.282 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-03-12T03:16:55.282 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [----------] Global test environment set-up. 2024-03-12T03:16:55.282 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (10927 ms) 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (2899 ms) 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (6547 ms) 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (3029 ms) 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3442 ms) 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3434 ms) 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-03-12T03:16:55.283 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (7871 ms) 2024-03-12T03:16:55.284 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (38149 ms total) 2024-03-12T03:16:55.284 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: 2024-03-12T03:16:55.284 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [----------] Global test environment tear-down 2024-03-12T03:16:55.284 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (38149 ms total) 2024-03-12T03:16:55.284 INFO:tasks.workunit.client.0.smithi094.stdout: write_operations: [ PASSED ] 7 tests. 2024-03-12T03:16:55.287 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: Running main() from gmock_main.cc 2024-03-12T03:16:55.287 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-03-12T03:16:55.287 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: [----------] Global test environment set-up. 2024-03-12T03:16:55.287 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-03-12T03:16:55.287 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-03-12T03:16:55.287 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (18420 ms) 2024-03-12T03:16:55.287 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-03-12T03:16:55.287 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (8491 ms) 2024-03-12T03:16:55.287 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-03-12T03:16:55.288 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (11281 ms) 2024-03-12T03:16:55.288 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: [----------] 3 tests from NeoradosECList (38192 ms total) 2024-03-12T03:16:55.288 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: 2024-03-12T03:16:55.288 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: [----------] Global test environment tear-down 2024-03-12T03:16:55.288 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (38192 ms total) 2024-03-12T03:16:55.288 INFO:tasks.workunit.client.0.smithi094.stdout: ec_list: [ PASSED ] 3 tests. 2024-03-12T03:16:55.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[40484]: pgmap v110: 720 pgs: 296 unknown, 424 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:16:55.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7"}]': finished 2024-03-12T03:16:55.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[40484]: from='client.28036 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi094-79350-3"}]': finished 2024-03-12T03:16:55.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[40484]: osdmap e103: 8 total, 8 up, 8 in 2024-03-12T03:16:55.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2775502452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:55.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2734822432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:55.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[40484]: from='client.46229 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:55.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[40484]: from='client.28036 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:55.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:55.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4"}]: dispatch 2024-03-12T03:16:55.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4"}]: dispatch 2024-03-12T03:16:55.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[29291]: pgmap v110: 720 pgs: 296 unknown, 424 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:16:55.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-7"}]': finished 2024-03-12T03:16:55.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[29291]: from='client.28036 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi094-79350-3"}]': finished 2024-03-12T03:16:55.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[29291]: osdmap e103: 8 total, 8 up, 8 in 2024-03-12T03:16:55.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2775502452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:55.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2734822432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:55.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[29291]: from='client.46229 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:55.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[29291]: from='client.28036 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi094-79350-3"}]: dispatch 2024-03-12T03:16:55.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:55.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4"}]: dispatch 2024-03-12T03:16:55.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:55 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4"}]: dispatch 2024-03-12T03:16:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[40484]: pgmap v113: 712 pgs: 32 creating+peering, 96 unknown, 584 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2024-03-12T03:16:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[40484]: from='client.46229 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[40484]: from='client.28036 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi094-79350-3"}]': finished 2024-03-12T03:16:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4"}]': finished 2024-03-12T03:16:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[40484]: osdmap e104: 8 total, 8 up, 8 in 2024-03-12T03:16:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1827800575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi094-78081-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[40484]: from='client.47321 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi094-78081-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6"}]: dispatch 2024-03-12T03:16:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6"}]: dispatch 2024-03-12T03:16:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[29291]: pgmap v113: 712 pgs: 32 creating+peering, 96 unknown, 584 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2024-03-12T03:16:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[29291]: from='client.46229 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[29291]: from='client.28036 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi094-79350-3"}]': finished 2024-03-12T03:16:56.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4"}]': finished 2024-03-12T03:16:56.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[29291]: osdmap e104: 8 total, 8 up, 8 in 2024-03-12T03:16:56.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1827800575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi094-78081-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:56.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[29291]: from='client.47321 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi094-78081-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:56.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6"}]: dispatch 2024-03-12T03:16:56.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6"}]: dispatch 2024-03-12T03:16:56.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:56 smithi151 ceph-mon[32923]: pgmap v113: 712 pgs: 32 creating+peering, 96 unknown, 584 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2024-03-12T03:16:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:56 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:56 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:56 smithi151 ceph-mon[32923]: from='client.46229 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi094-78099-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:56 smithi151 ceph-mon[32923]: from='client.28036 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi094-79350-3"}]': finished 2024-03-12T03:16:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:56 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4"}]': finished 2024-03-12T03:16:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:56 smithi151 ceph-mon[32923]: osdmap e104: 8 total, 8 up, 8 in 2024-03-12T03:16:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1827800575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi094-78081-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:56 smithi151 ceph-mon[32923]: from='client.47321 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi094-78081-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6"}]: dispatch 2024-03-12T03:16:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:56 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6"}]: dispatch 2024-03-12T03:16:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:16:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[29291]: from='client.47321 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi094-78081-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6"}]': finished 2024-03-12T03:16:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[29291]: osdmap e105: 8 total, 8 up, 8 in 2024-03-12T03:16:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1335012723' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi094-78644-6", "pool2": "test-rados-api-smithi094-78644-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi094-78644-6", "pool2": "test-rados-api-smithi094-78644-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[29291]: from='client.40402 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:16:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:16:57.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:57.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:57 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:16:57.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:57 smithi151 ceph-mon[32923]: from='client.47321 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi094-78081-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:57.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:57 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6"}]': finished 2024-03-12T03:16:57.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:57 smithi151 ceph-mon[32923]: osdmap e105: 8 total, 8 up, 8 in 2024-03-12T03:16:57.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1335012723' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi094-78644-6", "pool2": "test-rados-api-smithi094-78644-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:57 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi094-78644-6", "pool2": "test-rados-api-smithi094-78644-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:57 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:57 smithi151 ceph-mon[32923]: from='client.40402 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.966 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:57.966 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:16:57.966 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[40484]: from='client.47321 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi094-78081-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:57.966 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi094-78644-4", "tierpool": "test-rados-api-smithi094-78644-6"}]': finished 2024-03-12T03:16:57.966 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[40484]: osdmap e105: 8 total, 8 up, 8 in 2024-03-12T03:16:57.966 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.966 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1335012723' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.966 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1637113034' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi094-78644-6", "pool2": "test-rados-api-smithi094-78644-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.966 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi094-78644-6", "pool2": "test-rados-api-smithi094-78644-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.966 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:57.966 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:57 smithi094 ceph-mon[40484]: from='client.40402 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:58.131 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: flushed 2024-03-12T03:16:58.131 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3008 ms) 2024-03-12T03:16:58.131 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: trying... 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: handle_notify cookie 94294924410224 notify_id 403726925829 notifier_gid 35348 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: timed out 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: flushing 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: flushed 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3011 ms) 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: List watches 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: notify2 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: handle_notify cookie 94294924410224 notify_id 420906795010 notifier_gid 35348 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: notify2 done 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: watch_check 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: unwatch2 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: flushing 2024-03-12T03:16:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: done 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3082 ms) 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: List watches 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: notify2 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: handle_notify cookie 94294924410224 notify_id 429496729604 notifier_gid 35348 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: notify2 done 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: watch_check 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: unwatch2 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: flushing 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: done 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3031 ms) 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16506 ms total) 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-12T03:16:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (41266 ms total) 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (11153 ms) 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-12T03:16:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (6397 ms) 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (6077 ms) 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3454 ms) 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3432 ms) 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3864 ms) 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (6850 ms) 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (41228 ms total) 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (41228 ms total) 2024-03-12T03:16:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-12T03:16:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: pgmap v116: 616 pgs: 32 creating+peering, 128 unknown, 456 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-12T03:16:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi094-78644-6", "pool2": "test-rados-api-smithi094-78644-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: from='client.40402 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: osdmap e106: 8 total, 8 up, 8 in 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3877607372' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi094-78099-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: from='client.40954 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi094-78099-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: from='client.40954 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi094-78099-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: osdmap e107: 8 total, 8 up, 8 in 2024-03-12T03:16:58.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:58.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:58.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: pgmap v116: 616 pgs: 32 creating+peering, 128 unknown, 456 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-12T03:16:58.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:58.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi094-78644-6", "pool2": "test-rados-api-smithi094-78644-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:16:58.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:58.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: from='client.40402 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:58.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: osdmap e106: 8 total, 8 up, 8 in 2024-03-12T03:16:58.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3877607372' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi094-78099-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:58.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: from='client.40954 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi094-78099-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:58.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:58.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:58.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:58.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:58.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: from='client.40954 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi094-78099-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:58.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:58.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: osdmap e107: 8 total, 8 up, 8 in 2024-03-12T03:16:58.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:58.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:16:58 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: pgmap v116: 616 pgs: 32 creating+peering, 128 unknown, 456 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-12T03:16:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi094-78644-6", "pool2": "test-rados-api-smithi094-78644-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:16:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: from='client.40402 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: osdmap e106: 8 total, 8 up, 8 in 2024-03-12T03:16:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3877607372' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi094-78099-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: from='client.40954 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi094-78099-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:16:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:16:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:16:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:16:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: from='client.40954 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi094-78099-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:16:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:16:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: osdmap e107: 8 total, 8 up, 8 in 2024-03-12T03:16:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:16:58.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:16:58 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:17:00.535 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:17:00.536 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[29291]: osdmap e108: 8 total, 8 up, 8 in 2024-03-12T03:17:00.536 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-9"}]: dispatch 2024-03-12T03:17:00.536 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2916842356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi094-78081-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:00.536 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-9"}]: dispatch 2024-03-12T03:17:00.536 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[29291]: from='client.42070 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi094-78081-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:00.536 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:00.536 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-03-12T03:17:00.536 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-03-12T03:17:00.536 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-03-12T03:17:00.536 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:17:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:17:00.537 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:17:00.537 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[40484]: osdmap e108: 8 total, 8 up, 8 in 2024-03-12T03:17:00.537 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-9"}]: dispatch 2024-03-12T03:17:00.537 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2916842356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi094-78081-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:00.537 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-9"}]: dispatch 2024-03-12T03:17:00.537 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[40484]: from='client.42070 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi094-78081-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:00.537 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:00.537 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-03-12T03:17:00.537 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-03-12T03:17:00.537 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-03-12T03:17:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:00 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:17:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:00 smithi151 ceph-mon[32923]: osdmap e108: 8 total, 8 up, 8 in 2024-03-12T03:17:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-9"}]: dispatch 2024-03-12T03:17:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2916842356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi094-78081-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:00 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-9"}]: dispatch 2024-03-12T03:17:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:00 smithi151 ceph-mon[32923]: from='client.42070 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi094-78081-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-03-12T03:17:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-03-12T03:17:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-03-12T03:17:01.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: pgmap v119: 644 pgs: 2 active, 26 creating+activating, 99 creating+peering, 84 unknown, 433 active+clean; 145 MiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:01.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-03-12T03:17:01.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-03-12T03:17:01.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: 20.2 deep-scrub starts 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: 20.2 deep-scrub ok 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: 20.5 deep-scrub starts 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: 20.5 deep-scrub ok 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-9"}]': finished 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.42070 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi094-78081-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: osdmap e109: 8 total, 8 up, 8 in 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-9", "mode": "writeback"}]: dispatch 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2960604497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi094-78099-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-9", "mode": "writeback"}]: dispatch 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.51950 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi094-78099-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:17:01.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: 20.8 deep-scrub starts 2024-03-12T03:17:01.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: 20.8 deep-scrub ok 2024-03-12T03:17:01.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:17:01.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:17:01.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-9", "mode": "writeback"}]': finished 2024-03-12T03:17:01.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.51950 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi094-78099-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:01.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: osdmap e110: 8 total, 8 up, 8 in 2024-03-12T03:17:01.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2688591872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:01.714 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[29291]: from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: pgmap v119: 644 pgs: 2 active, 26 creating+activating, 99 creating+peering, 84 unknown, 433 active+clean; 145 MiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-03-12T03:17:01.715 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: 20.2 deep-scrub starts 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: 20.2 deep-scrub ok 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: 20.5 deep-scrub starts 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: 20.5 deep-scrub ok 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-9"}]': finished 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.42070 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi094-78081-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: osdmap e109: 8 total, 8 up, 8 in 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-9", "mode": "writeback"}]: dispatch 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2960604497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi094-78099-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-9", "mode": "writeback"}]: dispatch 2024-03-12T03:17:01.716 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.51950 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi094-78099-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:01.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:01.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:17:01.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:17:01.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: 20.8 deep-scrub starts 2024-03-12T03:17:01.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: 20.8 deep-scrub ok 2024-03-12T03:17:01.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:17:01.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:17:01.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-9", "mode": "writeback"}]': finished 2024-03-12T03:17:01.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.51950 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi094-78099-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:01.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: osdmap e110: 8 total, 8 up, 8 in 2024-03-12T03:17:01.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2688591872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:01.717 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:01 smithi094 ceph-mon[40484]: from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:01.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: pgmap v119: 644 pgs: 2 active, 26 creating+activating, 99 creating+peering, 84 unknown, 433 active+clean; 145 MiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:01.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-03-12T03:17:01.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-03-12T03:17:01.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-03-12T03:17:01.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-03-12T03:17:01.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-03-12T03:17:01.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-03-12T03:17:01.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-03-12T03:17:01.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-03-12T03:17:01.718 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: 20.2 deep-scrub starts 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: 20.2 deep-scrub ok 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: 20.5 deep-scrub starts 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: 20.5 deep-scrub ok 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-9"}]': finished 2024-03-12T03:17:01.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.42070 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi094-78081-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: osdmap e109: 8 total, 8 up, 8 in 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-9", "mode": "writeback"}]: dispatch 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2960604497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi094-78099-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-9", "mode": "writeback"}]: dispatch 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.51950 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi094-78099-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: 20.8 deep-scrub starts 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: 20.8 deep-scrub ok 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-9", "mode": "writeback"}]': finished 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.51950 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi094-78099-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:01.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: osdmap e110: 8 total, 8 up, 8 in 2024-03-12T03:17:01.721 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2688591872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:01.721 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:01 smithi151 ceph-mon[32923]: from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:02 smithi151 ceph-mon[32923]: 20.0 deep-scrub starts 2024-03-12T03:17:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:02 smithi151 ceph-mon[32923]: 20.0 deep-scrub ok 2024-03-12T03:17:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:02 smithi151 ceph-mon[32923]: 20.7 deep-scrub starts 2024-03-12T03:17:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:02 smithi151 ceph-mon[32923]: 20.7 deep-scrub ok 2024-03-12T03:17:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:02 smithi151 ceph-mon[32923]: 20.6 deep-scrub starts 2024-03-12T03:17:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:02 smithi151 ceph-mon[32923]: 20.6 deep-scrub ok 2024-03-12T03:17:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:02.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[40484]: 20.0 deep-scrub starts 2024-03-12T03:17:02.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[40484]: 20.0 deep-scrub ok 2024-03-12T03:17:02.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[40484]: 20.7 deep-scrub starts 2024-03-12T03:17:02.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[40484]: 20.7 deep-scrub ok 2024-03-12T03:17:02.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[40484]: 20.6 deep-scrub starts 2024-03-12T03:17:02.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[40484]: 20.6 deep-scrub ok 2024-03-12T03:17:02.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:02.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:02.726 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:02.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[29291]: 20.0 deep-scrub starts 2024-03-12T03:17:02.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[29291]: 20.0 deep-scrub ok 2024-03-12T03:17:02.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[29291]: 20.7 deep-scrub starts 2024-03-12T03:17:02.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[29291]: 20.7 deep-scrub ok 2024-03-12T03:17:02.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[29291]: 20.6 deep-scrub starts 2024-03-12T03:17:02.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[29291]: 20.6 deep-scrub ok 2024-03-12T03:17:02.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:02.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:02.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: 20.3 deep-scrub starts 2024-03-12T03:17:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: 20.3 deep-scrub ok 2024-03-12T03:17:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: pgmap v122: 612 pgs: 1 active, 6 creating+activating, 36 creating+peering, 144 unknown, 425 active+clean; 145 MiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: 20.1 deep-scrub starts 2024-03-12T03:17:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: 20.1 deep-scrub ok 2024-03-12T03:17:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: 20.9 deep-scrub starts 2024-03-12T03:17:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: 20.9 deep-scrub ok 2024-03-12T03:17:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: 20.4 deep-scrub starts 2024-03-12T03:17:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: 20.4 deep-scrub ok 2024-03-12T03:17:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:03.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:03.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:03.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: osdmap e111: 8 total, 8 up, 8 in 2024-03-12T03:17:03.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:03.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/888463515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi094-78081-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:03.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:03.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: from='client.45580 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi094-78081-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:03.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:03.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:17:03.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: 20.3 deep-scrub starts 2024-03-12T03:17:03.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: 20.3 deep-scrub ok 2024-03-12T03:17:03.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: pgmap v122: 612 pgs: 1 active, 6 creating+activating, 36 creating+peering, 144 unknown, 425 active+clean; 145 MiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:03.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: 20.1 deep-scrub starts 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: 20.1 deep-scrub ok 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: 20.9 deep-scrub starts 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: 20.9 deep-scrub ok 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: 20.4 deep-scrub starts 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: 20.4 deep-scrub ok 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: osdmap e111: 8 total, 8 up, 8 in 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/888463515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi094-78081-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:03.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: from='client.45580 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi094-78081-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:03.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:03.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:03 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:17:03.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: 20.3 deep-scrub starts 2024-03-12T03:17:03.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: 20.3 deep-scrub ok 2024-03-12T03:17:03.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: pgmap v122: 612 pgs: 1 active, 6 creating+activating, 36 creating+peering, 144 unknown, 425 active+clean; 145 MiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:03.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: 20.1 deep-scrub starts 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: 20.1 deep-scrub ok 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: 20.9 deep-scrub starts 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: 20.9 deep-scrub ok 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: 20.4 deep-scrub starts 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: 20.4 deep-scrub ok 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78374-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: osdmap e111: 8 total, 8 up, 8 in 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/888463515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi094-78081-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: from='client.45580 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi094-78081-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:03.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:03.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:03 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:17:04.386 INFO:tasks.workunit.client.0.smithi094.stdout: misc: Running main() from gmock_main.cc 2024-03-12T03:17:04.386 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-03-12T03:17:04.386 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [----------] Global test environment set-up. 2024-03-12T03:17:04.387 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-03-12T03:17:04.387 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-03-12T03:17:04.387 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ OK ] NeoRadosMisc.Version (10035 ms) 2024-03-12T03:17:04.387 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-03-12T03:17:04.387 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (1751 ms) 2024-03-12T03:17:04.387 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-03-12T03:17:04.387 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ OK ] NeoRadosMisc.LongName (5564 ms) 2024-03-12T03:17:04.387 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-03-12T03:17:04.387 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (4332 ms) 2024-03-12T03:17:04.387 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-03-12T03:17:04.387 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3078 ms) 2024-03-12T03:17:04.387 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3206 ms) 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ OK ] NeoRadosMisc.Exec (3748 ms) 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3422 ms) 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3062 ms) 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3826 ms) 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2095 ms) 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3175 ms) 2024-03-12T03:17:04.388 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [----------] 12 tests from NeoRadosMisc (47294 ms total) 2024-03-12T03:17:04.389 INFO:tasks.workunit.client.0.smithi094.stdout: misc: 2024-03-12T03:17:04.389 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [----------] Global test environment tear-down 2024-03-12T03:17:04.389 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [==========] 12 tests from 1 test suite ran. (47294 ms total) 2024-03-12T03:17:04.389 INFO:tasks.workunit.client.0.smithi094.stdout: misc: [ PASSED ] 12 tests. 2024-03-12T03:17:04.397 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-12T03:17:04.397 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [----------] Global test environment set-up. 2024-03-12T03:17:04.397 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-12T03:17:04.397 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-12T03:17:04.397 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-12T03:17:04.397 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-12T03:17:04.397 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: 2024-03-12T03:17:04.397 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-12T03:17:04.397 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-12T03:17:04.397 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: 2024-03-12T03:16:16.819+0000 7eff4db7f780 -1 failed for service _ceph-mon._tcp 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: 2024-03-12T03:16:16.819+0000 7eff4db7f780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (67 ms) 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2011 ms) 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2078 ms total) 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: started 0x55ebe9af2600 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: started 0x55ebe9e24000 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: started 2 aios 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: waiting 0x55ebe9af2600 2024-03-12T03:17:04.398 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: waiting 0x55ebe9e24000 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: done. 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (12103 ms) 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (12103 ms total) 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ OK ] LibRadosMisc.Exec (156 ms) 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (5 ms) 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3 ms) 2024-03-12T03:17:04.399 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-12T03:17:04.400 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ OK ] LibRadosMisc.Applications (6920 ms) 2024-03-12T03:17:04.400 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-12T03:17:04.400 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-12T03:17:04.400 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-12T03:17:04.400 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-12T03:17:04.400 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-12T03:17:04.400 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (22266 ms) 2024-03-12T03:17:04.400 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [----------] 8 tests from LibRadosMisc (29350 ms total) 2024-03-12T03:17:04.400 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: 2024-03-12T03:17:04.400 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [----------] Global test environment tear-down 2024-03-12T03:17:04.400 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (47631 ms total) 2024-03-12T03:17:04.400 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-12T03:17:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: pgmap v125: 548 pgs: 1 active, 6 creating+activating, 20 creating+peering, 128 unknown, 393 active+clean; 145 MiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.45643 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.45580 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi094-78081-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi094-78434-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9"}]: dispatch 2024-03-12T03:17:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2688591872' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache", "force_nonempty":""}]: dispatch 2024-03-12T03:17:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:17:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi094-78434-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9"}]: dispatch 2024-03-12T03:17:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache", "force_nonempty":""}]: dispatch 2024-03-12T03:17:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:17:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:17:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:17:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9"}]': finished 2024-03-12T03:17:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache", "force_nonempty":""}]': finished 2024-03-12T03:17:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2688591872' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78374-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3310965694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi094-78099-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:04.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: pgmap v125: 548 pgs: 1 active, 6 creating+activating, 20 creating+peering, 128 unknown, 393 active+clean; 145 MiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:04.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.45643 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:04.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.45580 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi094-78081-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:04.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi094-78434-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:04.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9"}]: dispatch 2024-03-12T03:17:04.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2688591872' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache", "force_nonempty":""}]: dispatch 2024-03-12T03:17:04.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:04.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:17:04.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi094-78434-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:04.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9"}]: dispatch 2024-03-12T03:17:04.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache", "force_nonempty":""}]: dispatch 2024-03-12T03:17:04.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:17:04.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:17:04.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:17:04.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9"}]': finished 2024-03-12T03:17:04.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache", "force_nonempty":""}]': finished 2024-03-12T03:17:04.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2688591872' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78374-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:04.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3310965694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi094-78099-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:04.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:04.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: osdmap e113: 8 total, 8 up, 8 in 2024-03-12T03:17:04.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78374-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:04.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[40484]: from='client.46903 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi094-78099-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:04.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: osdmap e113: 8 total, 8 up, 8 in 2024-03-12T03:17:04.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78374-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:04.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:04 smithi094 ceph-mon[29291]: from='client.46903 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi094-78099-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: pgmap v125: 548 pgs: 1 active, 6 creating+activating, 20 creating+peering, 128 unknown, 393 active+clean; 145 MiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.45643 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.45580 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi094-78081-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi094-78434-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9"}]: dispatch 2024-03-12T03:17:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2688591872' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache", "force_nonempty":""}]: dispatch 2024-03-12T03:17:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:17:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi094-78434-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9"}]: dispatch 2024-03-12T03:17:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache", "force_nonempty":""}]: dispatch 2024-03-12T03:17:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:17:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:17:04.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:17:04.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-9"}]': finished 2024-03-12T03:17:04.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache", "force_nonempty":""}]': finished 2024-03-12T03:17:04.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2688591872' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78374-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:04.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3310965694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi094-78099-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:04.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:04.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: osdmap e113: 8 total, 8 up, 8 in 2024-03-12T03:17:04.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78374-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:04.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:04 smithi151 ceph-mon[32923]: from='client.46903 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi094-78099-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[29291]: pgmap v128: 548 pgs: 96 unknown, 32 creating+peering, 420 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 2 op/s 2024-03-12T03:17:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[29291]: from='client.45643 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi094-78434-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]': finished 2024-03-12T03:17:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[29291]: from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78374-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[29291]: from='client.46903 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi094-78099-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[29291]: osdmap e114: 8 total, 8 up, 8 in 2024-03-12T03:17:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2688591872' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache"}]: dispatch 2024-03-12T03:17:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3800314369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi094-78081-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[29291]: from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache"}]: dispatch 2024-03-12T03:17:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[29291]: from='client.56453 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi094-78081-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:06.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[40484]: pgmap v128: 548 pgs: 96 unknown, 32 creating+peering, 420 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 2 op/s 2024-03-12T03:17:06.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[40484]: from='client.45643 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi094-78434-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]': finished 2024-03-12T03:17:06.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[40484]: from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78374-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:06.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[40484]: from='client.46903 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi094-78099-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:06.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[40484]: osdmap e114: 8 total, 8 up, 8 in 2024-03-12T03:17:06.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2688591872' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache"}]: dispatch 2024-03-12T03:17:06.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3800314369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi094-78081-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:06.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:06.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[40484]: from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache"}]: dispatch 2024-03-12T03:17:06.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:06 smithi094 ceph-mon[40484]: from='client.56453 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi094-78081-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:06.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:06 smithi151 ceph-mon[32923]: pgmap v128: 548 pgs: 96 unknown, 32 creating+peering, 420 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 2 op/s 2024-03-12T03:17:06.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:06 smithi151 ceph-mon[32923]: from='client.45643 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi094-78434-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]': finished 2024-03-12T03:17:06.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:06 smithi151 ceph-mon[32923]: from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78374-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:06.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:06 smithi151 ceph-mon[32923]: from='client.46903 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi094-78099-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:06.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:06 smithi151 ceph-mon[32923]: osdmap e114: 8 total, 8 up, 8 in 2024-03-12T03:17:06.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2688591872' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache"}]: dispatch 2024-03-12T03:17:06.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3800314369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi094-78081-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:06.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:06.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:06 smithi151 ceph-mon[32923]: from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache"}]: dispatch 2024-03-12T03:17:06.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:06 smithi151 ceph-mon[32923]: from='client.56453 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi094-78081-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:07.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:07 smithi094 ceph-mon[40484]: from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache"}]': finished 2024-03-12T03:17:07.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:07 smithi094 ceph-mon[40484]: from='client.56453 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi094-78081-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:07.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:07 smithi094 ceph-mon[40484]: osdmap e115: 8 total, 8 up, 8 in 2024-03-12T03:17:07.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:07.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:07 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:07.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:07 smithi094 ceph-mon[29291]: from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache"}]': finished 2024-03-12T03:17:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:07 smithi094 ceph-mon[29291]: from='client.56453 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi094-78081-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:07 smithi094 ceph-mon[29291]: osdmap e115: 8 total, 8 up, 8 in 2024-03-12T03:17:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:07 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:07.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:07 smithi151 ceph-mon[32923]: from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78374-10", "tierpool":"test-rados-api-smithi094-78374-10-cache"}]': finished 2024-03-12T03:17:07.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:07 smithi151 ceph-mon[32923]: from='client.56453 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi094-78081-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:07.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:07 smithi151 ceph-mon[32923]: osdmap e115: 8 total, 8 up, 8 in 2024-03-12T03:17:07.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:07.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:07 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:07.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:07.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:17:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:17:08.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: Running main() from gmock_main.cc 2024-03-12T03:17:08.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-12T03:17:08.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [----------] Global test environment set-up. 2024-03-12T03:17:08.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-12T03:17:08.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-12T03:17:08.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ OK ] LibRadosPools.PoolList (10352 ms) 2024-03-12T03:17:08.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-12T03:17:08.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3826 ms) 2024-03-12T03:17:08.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-12T03:17:08.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (5762 ms) 2024-03-12T03:17:08.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2820 ms) 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3059 ms) 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (7410 ms) 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6091 ms) 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4133 ms) 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7924 ms) 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [----------] 9 tests from LibRadosPools (51378 ms total) 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [----------] Global test environment tear-down 2024-03-12T03:17:08.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (51378 ms total) 2024-03-12T03:17:08.152 INFO:tasks.workunit.client.0.smithi094.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-12T03:17:08.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:08 smithi094 ceph-mon[29291]: pgmap v131: 556 pgs: 136 unknown, 32 creating+peering, 388 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 18 MiB/s wr, 4 op/s 2024-03-12T03:17:08.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:08 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:08.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:08 smithi094 ceph-mon[29291]: osdmap e116: 8 total, 8 up, 8 in 2024-03-12T03:17:08.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1483805417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:08.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:08 smithi094 ceph-mon[29291]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:08.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:08 smithi094 ceph-mon[40484]: pgmap v131: 556 pgs: 136 unknown, 32 creating+peering, 388 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 18 MiB/s wr, 4 op/s 2024-03-12T03:17:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:08 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:08 smithi094 ceph-mon[40484]: osdmap e116: 8 total, 8 up, 8 in 2024-03-12T03:17:08.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1483805417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:08.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:08 smithi094 ceph-mon[40484]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:08.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:08.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:08 smithi151 ceph-mon[32923]: pgmap v131: 556 pgs: 136 unknown, 32 creating+peering, 388 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 18 MiB/s wr, 4 op/s 2024-03-12T03:17:08.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:08 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:08.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:08 smithi151 ceph-mon[32923]: osdmap e116: 8 total, 8 up, 8 in 2024-03-12T03:17:08.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1483805417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:08.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:08 smithi151 ceph-mon[32923]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:08.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[29291]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[29291]: osdmap e117: 8 total, 8 up, 8 in 2024-03-12T03:17:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3209362343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi094-78081-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[29291]: from='client.56471 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi094-78081-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:17:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:17:09.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[29291]: from='client.56471 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi094-78081-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:09.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:17:09.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[29291]: osdmap e118: 8 total, 8 up, 8 in 2024-03-12T03:17:09.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-11"}]: dispatch 2024-03-12T03:17:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[40484]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[40484]: osdmap e117: 8 total, 8 up, 8 in 2024-03-12T03:17:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3209362343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi094-78081-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[40484]: from='client.56471 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi094-78081-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:17:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:17:09.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[40484]: from='client.56471 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi094-78081-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:09.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:17:09.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[40484]: osdmap e118: 8 total, 8 up, 8 in 2024-03-12T03:17:09.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-11"}]: dispatch 2024-03-12T03:17:09.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:09.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:09 smithi151 ceph-mon[32923]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:09.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:09 smithi151 ceph-mon[32923]: osdmap e117: 8 total, 8 up, 8 in 2024-03-12T03:17:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3209362343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi094-78081-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:09 smithi151 ceph-mon[32923]: from='client.56471 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi094-78081-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:17:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:09 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:17:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:09 smithi151 ceph-mon[32923]: from='client.56471 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi094-78081-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:09 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:17:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:09 smithi151 ceph-mon[32923]: osdmap e118: 8 total, 8 up, 8 in 2024-03-12T03:17:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-11"}]: dispatch 2024-03-12T03:17:10.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[29291]: pgmap v134: 492 pgs: 32 creating+peering, 64 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 3.5 KiB/s wr, 4 op/s 2024-03-12T03:17:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[29291]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-11"}]: dispatch 2024-03-12T03:17:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-11"}]': finished 2024-03-12T03:17:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-11", "mode": "writeback"}]: dispatch 2024-03-12T03:17:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[29291]: osdmap e119: 8 total, 8 up, 8 in 2024-03-12T03:17:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-11", "mode": "writeback"}]: dispatch 2024-03-12T03:17:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3405782239' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi094-78099-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[29291]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi094-78099-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[40484]: pgmap v134: 492 pgs: 32 creating+peering, 64 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 3.5 KiB/s wr, 4 op/s 2024-03-12T03:17:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[40484]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-11"}]: dispatch 2024-03-12T03:17:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-11"}]': finished 2024-03-12T03:17:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-11", "mode": "writeback"}]: dispatch 2024-03-12T03:17:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[40484]: osdmap e119: 8 total, 8 up, 8 in 2024-03-12T03:17:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-11", "mode": "writeback"}]: dispatch 2024-03-12T03:17:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3405782239' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi094-78099-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-mon[40484]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi094-78099-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:10.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:10 smithi151 ceph-mon[32923]: pgmap v134: 492 pgs: 32 creating+peering, 64 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 3.5 KiB/s wr, 4 op/s 2024-03-12T03:17:10.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:10 smithi151 ceph-mon[32923]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:10.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:10 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-11"}]: dispatch 2024-03-12T03:17:10.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:10.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:10 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-11"}]': finished 2024-03-12T03:17:10.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-11", "mode": "writeback"}]: dispatch 2024-03-12T03:17:10.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:10 smithi151 ceph-mon[32923]: osdmap e119: 8 total, 8 up, 8 in 2024-03-12T03:17:10.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:10 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-11", "mode": "writeback"}]: dispatch 2024-03-12T03:17:10.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3405782239' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi094-78099-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:10.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:10 smithi151 ceph-mon[32923]: from='client.48652 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi094-78099-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:10.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:17:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:17:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:17:11.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:17:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-11", "mode": "writeback"}]': finished 2024-03-12T03:17:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[29291]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi094-78099-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[29291]: osdmap e120: 8 total, 8 up, 8 in 2024-03-12T03:17:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2495726322' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi094-78081-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:11.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[29291]: from='client.48658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi094-78081-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:11.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:11.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:17:11.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-11", "mode": "writeback"}]': finished 2024-03-12T03:17:11.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[40484]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi094-78099-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:11.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[40484]: osdmap e120: 8 total, 8 up, 8 in 2024-03-12T03:17:11.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2495726322' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi094-78081-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:11.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:11 smithi094 ceph-mon[40484]: from='client.48658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi094-78081-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:11.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:11.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:11 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:17:11.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:11 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-11", "mode": "writeback"}]': finished 2024-03-12T03:17:11.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:11 smithi151 ceph-mon[32923]: from='client.48652 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi094-78099-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:11.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:11 smithi151 ceph-mon[32923]: osdmap e120: 8 total, 8 up, 8 in 2024-03-12T03:17:11.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2495726322' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi094-78081-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:11.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:11 smithi151 ceph-mon[32923]: from='client.48658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi094-78081-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:12.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:12 smithi094 ceph-mon[29291]: pgmap v137: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-12T03:17:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:12 smithi094 ceph-mon[29291]: from='client.48658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi094-78081-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:12 smithi094 ceph-mon[29291]: osdmap e121: 8 total, 8 up, 8 in 2024-03-12T03:17:12.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:12 smithi094 ceph-mon[40484]: pgmap v137: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-12T03:17:12.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:12.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:12 smithi094 ceph-mon[40484]: from='client.48658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi094-78081-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:12.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:12 smithi094 ceph-mon[40484]: osdmap e121: 8 total, 8 up, 8 in 2024-03-12T03:17:12.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:12 smithi151 ceph-mon[32923]: pgmap v137: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-12T03:17:12.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:12.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:12 smithi151 ceph-mon[32923]: from='client.48658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi094-78081-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:12.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:12 smithi151 ceph-mon[32923]: osdmap e121: 8 total, 8 up, 8 in 2024-03-12T03:17:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:17:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[29291]: osdmap e122: 8 total, 8 up, 8 in 2024-03-12T03:17:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11"}]: dispatch 2024-03-12T03:17:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1142566352' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi094-78099-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11"}]: dispatch 2024-03-12T03:17:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[29291]: from='client.48667 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi094-78099-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:13.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:13.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:13.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:13.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:17:13.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[40484]: osdmap e122: 8 total, 8 up, 8 in 2024-03-12T03:17:13.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11"}]: dispatch 2024-03-12T03:17:13.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1142566352' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi094-78099-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:13.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11"}]: dispatch 2024-03-12T03:17:13.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:13 smithi094 ceph-mon[40484]: from='client.48667 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi094-78099-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:13.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:13.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:13.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:13 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:13.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:13 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:17:13.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:13 smithi151 ceph-mon[32923]: osdmap e122: 8 total, 8 up, 8 in 2024-03-12T03:17:13.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11"}]: dispatch 2024-03-12T03:17:13.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1142566352' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi094-78099-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:13.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:13 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11"}]: dispatch 2024-03-12T03:17:13.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:13 smithi151 ceph-mon[32923]: from='client.48667 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi094-78099-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:14.131 INFO:tasks.workunit.client.0.smithi094.stdout: io: Running main() from gmock_main.cc 2024-03-12T03:17:14.131 INFO:tasks.workunit.client.0.smithi094.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-03-12T03:17:14.131 INFO:tasks.workunit.client.0.smithi094.stdout: io: [----------] Global test environment set-up. 2024-03-12T03:17:14.131 INFO:tasks.workunit.client.0.smithi094.stdout: io: [----------] 14 tests from NeoRadosIo 2024-03-12T03:17:14.131 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-03-12T03:17:14.131 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.Limits (12859 ms) 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (4542 ms) 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.ReadOp (4323 ms) 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.SparseRead (3781 ms) 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3483 ms) 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3936 ms) 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3274 ms) 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3295 ms) 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-03-12T03:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2562 ms) 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.Trunc (3303 ms) 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.Remove (2730 ms) 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3021 ms) 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.RmXattr (3013 ms) 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2962 ms) 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [----------] 14 tests from NeoRadosIo (57084 ms total) 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [----------] Global test environment tear-down 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [==========] 14 tests from 1 test suite ran. (57084 ms total) 2024-03-12T03:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: io: [ PASSED ] 14 tests. 2024-03-12T03:17:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[29291]: pgmap v140: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:17:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11"}]': finished 2024-03-12T03:17:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[29291]: from='client.48667 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi094-78099-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[29291]: osdmap e123: 8 total, 8 up, 8 in 2024-03-12T03:17:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2605516142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi094-78081-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[40484]: pgmap v140: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:17:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11"}]': finished 2024-03-12T03:17:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[40484]: from='client.48667 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi094-78099-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[40484]: osdmap e123: 8 total, 8 up, 8 in 2024-03-12T03:17:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2605516142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi094-78081-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:14 smithi151 ceph-mon[32923]: pgmap v140: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:14 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:17:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:14 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-11"}]': finished 2024-03-12T03:17:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:14 smithi151 ceph-mon[32923]: from='client.48667 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi094-78099-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:14 smithi151 ceph-mon[32923]: osdmap e123: 8 total, 8 up, 8 in 2024-03-12T03:17:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2605516142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi094-78081-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:14.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:16.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:16 smithi094 ceph-mon[29291]: pgmap v143: 492 pgs: 64 unknown, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 261 KiB/s wr, 12 op/s 2024-03-12T03:17:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2605516142' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi094-78081-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:16 smithi094 ceph-mon[29291]: osdmap e124: 8 total, 8 up, 8 in 2024-03-12T03:17:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:16 smithi094 ceph-mon[29291]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:16.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:16 smithi094 ceph-mon[40484]: pgmap v143: 492 pgs: 64 unknown, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 261 KiB/s wr, 12 op/s 2024-03-12T03:17:16.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2605516142' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi094-78081-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:16.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:16 smithi094 ceph-mon[40484]: osdmap e124: 8 total, 8 up, 8 in 2024-03-12T03:17:16.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:16.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:16.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:16 smithi094 ceph-mon[40484]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:16.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:16 smithi151 ceph-mon[32923]: pgmap v143: 492 pgs: 64 unknown, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 261 KiB/s wr, 12 op/s 2024-03-12T03:17:16.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2605516142' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi094-78081-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:16.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:16 smithi151 ceph-mon[32923]: osdmap e124: 8 total, 8 up, 8 in 2024-03-12T03:17:16.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:16.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:16.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:16 smithi151 ceph-mon[32923]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[29291]: from='client.45643 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]': finished 2024-03-12T03:17:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[29291]: osdmap e125: 8 total, 8 up, 8 in 2024-03-12T03:17:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/957819259' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi094-78099-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[29291]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[29291]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi094-78099-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:17.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[40484]: from='client.45643 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]': finished 2024-03-12T03:17:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[40484]: osdmap e125: 8 total, 8 up, 8 in 2024-03-12T03:17:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/957819259' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi094-78099-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[40484]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[40484]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi094-78099-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:17 smithi151 ceph-mon[32923]: from='client.45643 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi094-78434-10"}]': finished 2024-03-12T03:17:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:17 smithi151 ceph-mon[32923]: osdmap e125: 8 total, 8 up, 8 in 2024-03-12T03:17:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/760958412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/957819259' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi094-78099-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:17 smithi151 ceph-mon[32923]: from='client.45643 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]: dispatch 2024-03-12T03:17:17.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:17 smithi151 ceph-mon[32923]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi094-78099-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:17.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:17 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:17.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:17.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:17:17 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:17:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: pgmap v146: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 257 KiB/s wr, 3 op/s 2024-03-12T03:17:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: from='client.45643 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]': finished 2024-03-12T03:17:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi094-78099-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: osdmap e126: 8 total, 8 up, 8 in 2024-03-12T03:17:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/18197909' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi094-78081-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: from='client.56507 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi094-78081-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:18.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: from='client.56507 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi094-78081-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:18.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:18.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: osdmap e127: 8 total, 8 up, 8 in 2024-03-12T03:17:18.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi094-78434-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: pgmap v146: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 257 KiB/s wr, 3 op/s 2024-03-12T03:17:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: from='client.45643 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]': finished 2024-03-12T03:17:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi094-78099-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: osdmap e126: 8 total, 8 up, 8 in 2024-03-12T03:17:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/18197909' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi094-78081-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: from='client.56507 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi094-78081-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:18.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:18.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:18.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:18.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:18.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: from='client.56507 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi094-78081-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:18.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:18.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: osdmap e127: 8 total, 8 up, 8 in 2024-03-12T03:17:18.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi094-78434-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:18.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: pgmap v146: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 257 KiB/s wr, 3 op/s 2024-03-12T03:17:18.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: from='client.45643 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi094-78434-10"}]': finished 2024-03-12T03:17:18.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi094-78099-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: osdmap e126: 8 total, 8 up, 8 in 2024-03-12T03:17:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/18197909' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi094-78081-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: from='client.56507 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi094-78081-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: from='client.56507 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi094-78081-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: osdmap e127: 8 total, 8 up, 8 in 2024-03-12T03:17:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi094-78434-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:19 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:17:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:17:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:19 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:17:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:17:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:19 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:17:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:17:20.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:17:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:17:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[40484]: pgmap v149: 452 pgs: 32 creating+peering, 420 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 277 KiB/s wr, 20 op/s 2024-03-12T03:17:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[40484]: osdmap e128: 8 total, 8 up, 8 in 2024-03-12T03:17:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4292540132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi094-78099-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi094-78434-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]': finished 2024-03-12T03:17:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4292540132' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi094-78099-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[40484]: osdmap e129: 8 total, 8 up, 8 in 2024-03-12T03:17:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/276066437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi094-78081-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[40484]: from='client.56522 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi094-78081-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[29291]: pgmap v149: 452 pgs: 32 creating+peering, 420 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 277 KiB/s wr, 20 op/s 2024-03-12T03:17:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[29291]: osdmap e128: 8 total, 8 up, 8 in 2024-03-12T03:17:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4292540132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi094-78099-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi094-78434-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]': finished 2024-03-12T03:17:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4292540132' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi094-78099-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:20.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[29291]: osdmap e129: 8 total, 8 up, 8 in 2024-03-12T03:17:20.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/276066437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi094-78081-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:20.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[29291]: from='client.56522 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi094-78081-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:20.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:20 smithi151 ceph-mon[32923]: pgmap v149: 452 pgs: 32 creating+peering, 420 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 277 KiB/s wr, 20 op/s 2024-03-12T03:17:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:20 smithi151 ceph-mon[32923]: osdmap e128: 8 total, 8 up, 8 in 2024-03-12T03:17:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:20 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4292540132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi094-78099-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi094-78434-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]': finished 2024-03-12T03:17:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4292540132' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi094-78099-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:20 smithi151 ceph-mon[32923]: osdmap e129: 8 total, 8 up, 8 in 2024-03-12T03:17:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/276066437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi094-78081-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:20 smithi151 ceph-mon[32923]: from='client.56522 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi094-78081-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:22.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:22 smithi094 ceph-mon[40484]: pgmap v152: 492 pgs: 72 unknown, 420 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 282 KiB/s wr, 27 op/s 2024-03-12T03:17:22.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:22 smithi094 ceph-mon[40484]: from='client.56522 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi094-78081-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:22.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:22 smithi094 ceph-mon[40484]: osdmap e130: 8 total, 8 up, 8 in 2024-03-12T03:17:22.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:17:22.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:22.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:22 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:17:22.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:22 smithi094 ceph-mon[29291]: pgmap v152: 492 pgs: 72 unknown, 420 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 282 KiB/s wr, 27 op/s 2024-03-12T03:17:22.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:22 smithi094 ceph-mon[29291]: from='client.56522 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi094-78081-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:22.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:22 smithi094 ceph-mon[29291]: osdmap e130: 8 total, 8 up, 8 in 2024-03-12T03:17:22.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:17:22.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:22.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:22 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:17:22.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:22 smithi151 ceph-mon[32923]: pgmap v152: 492 pgs: 72 unknown, 420 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 282 KiB/s wr, 27 op/s 2024-03-12T03:17:22.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:22 smithi151 ceph-mon[32923]: from='client.56522 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi094-78081-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:22.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:22 smithi151 ceph-mon[32923]: osdmap e130: 8 total, 8 up, 8 in 2024-03-12T03:17:22.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:17:22.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:22.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:22 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:17:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:17:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: osdmap e131: 8 total, 8 up, 8 in 2024-03-12T03:17:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-13"}]: dispatch 2024-03-12T03:17:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-13"}]: dispatch 2024-03-12T03:17:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2319382332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi094-78099-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi094-78099-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-13"}]': finished 2024-03-12T03:17:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi094-78099-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: osdmap e132: 8 total, 8 up, 8 in 2024-03-12T03:17:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/757173553' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi094-78081-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-13", "mode": "writeback"}]: dispatch 2024-03-12T03:17:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi094-78081-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-13", "mode": "writeback"}]: dispatch 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: osdmap e131: 8 total, 8 up, 8 in 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-13"}]: dispatch 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-13"}]: dispatch 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2319382332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi094-78099-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi094-78099-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-13"}]': finished 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi094-78099-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: osdmap e132: 8 total, 8 up, 8 in 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/757173553' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi094-78081-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-13", "mode": "writeback"}]: dispatch 2024-03-12T03:17:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi094-78081-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:23.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:23 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-13", "mode": "writeback"}]: dispatch 2024-03-12T03:17:23.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:17:23.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: osdmap e131: 8 total, 8 up, 8 in 2024-03-12T03:17:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-13"}]: dispatch 2024-03-12T03:17:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-13"}]: dispatch 2024-03-12T03:17:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2319382332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi094-78099-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi094-78099-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-13"}]': finished 2024-03-12T03:17:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi094-78099-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: osdmap e132: 8 total, 8 up, 8 in 2024-03-12T03:17:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/757173553' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi094-78081-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-13", "mode": "writeback"}]: dispatch 2024-03-12T03:17:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi094-78081-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:23 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-13", "mode": "writeback"}]: dispatch 2024-03-12T03:17:24.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:24 smithi094 ceph-mon[40484]: pgmap v155: 460 pgs: 72 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:24 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:24 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:17:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:24 smithi094 ceph-mon[40484]: from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi094-78081-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:24 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-13", "mode": "writeback"}]': finished 2024-03-12T03:17:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:24 smithi094 ceph-mon[40484]: osdmap e133: 8 total, 8 up, 8 in 2024-03-12T03:17:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:24 smithi094 ceph-mon[29291]: pgmap v155: 460 pgs: 72 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:24 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:24 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:17:24.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:24 smithi094 ceph-mon[29291]: from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi094-78081-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:24.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:24 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-13", "mode": "writeback"}]': finished 2024-03-12T03:17:24.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:24 smithi094 ceph-mon[29291]: osdmap e133: 8 total, 8 up, 8 in 2024-03-12T03:17:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:24 smithi151 ceph-mon[32923]: pgmap v155: 460 pgs: 72 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:24 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:24 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:17:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:24 smithi151 ceph-mon[32923]: from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi094-78081-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:24 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-13", "mode": "writeback"}]': finished 2024-03-12T03:17:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:24 smithi151 ceph-mon[32923]: osdmap e133: 8 total, 8 up, 8 in 2024-03-12T03:17:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-03-12T03:17:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-03-12T03:17:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-03-12T03:17:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-03-12T03:17:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-03-12T03:17:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-03-12T03:17:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-03-12T03:17:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-03-12T03:17:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-03-12T03:17:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-03-12T03:17:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: osdmap e134: 8 total, 8 up, 8 in 2024-03-12T03:17:25.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/717075236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi094-78099-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:25.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[40484]: from='client.48727 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi094-78099-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:25.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:25.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-03-12T03:17:25.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-03-12T03:17:25.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-03-12T03:17:25.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-03-12T03:17:25.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-03-12T03:17:25.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-03-12T03:17:25.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-03-12T03:17:25.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-03-12T03:17:25.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-03-12T03:17:25.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-03-12T03:17:25.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: osdmap e134: 8 total, 8 up, 8 in 2024-03-12T03:17:25.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/717075236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi094-78099-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:25.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:25 smithi094 ceph-mon[29291]: from='client.48727 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi094-78099-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-03-12T03:17:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-03-12T03:17:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-03-12T03:17:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-03-12T03:17:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-03-12T03:17:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-03-12T03:17:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-03-12T03:17:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-03-12T03:17:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-03-12T03:17:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-03-12T03:17:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: osdmap e134: 8 total, 8 up, 8 in 2024-03-12T03:17:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/717075236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi094-78099-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:25 smithi151 ceph-mon[32923]: from='client.48727 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi094-78099-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: pgmap v158: 460 pgs: 32 unknown, 32 creating+peering, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 101 op/s 2024-03-12T03:17:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-03-12T03:17:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-03-12T03:17:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-03-12T03:17:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-03-12T03:17:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-03-12T03:17:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-03-12T03:17:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-03-12T03:17:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-03-12T03:17:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-03-12T03:17:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-03-12T03:17:26.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: 159.3 scrub starts 2024-03-12T03:17:26.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: 159.3 scrub ok 2024-03-12T03:17:26.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: 159.0 deep-scrub starts 2024-03-12T03:17:26.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: 159.0 deep-scrub ok 2024-03-12T03:17:26.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:26.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:26.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='client.48727 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi094-78099-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:26.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: osdmap e135: 8 total, 8 up, 8 in 2024-03-12T03:17:26.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1899314716' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi094-78081-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:26.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[29291]: from='client.48736 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi094-78081-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:26.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: pgmap v158: 460 pgs: 32 unknown, 32 creating+peering, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 101 op/s 2024-03-12T03:17:26.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: 159.3 scrub starts 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: 159.3 scrub ok 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: 159.0 deep-scrub starts 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: 159.0 deep-scrub ok 2024-03-12T03:17:26.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:26.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:26.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='client.48727 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi094-78099-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:26.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: osdmap e135: 8 total, 8 up, 8 in 2024-03-12T03:17:26.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1899314716' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi094-78081-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:26.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:26 smithi094 ceph-mon[40484]: from='client.48736 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi094-78081-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: pgmap v158: 460 pgs: 32 unknown, 32 creating+peering, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 101 op/s 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: 159.3 scrub starts 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: 159.3 scrub ok 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: 159.0 deep-scrub starts 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: 159.0 deep-scrub ok 2024-03-12T03:17:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='client.48727 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi094-78099-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: osdmap e135: 8 total, 8 up, 8 in 2024-03-12T03:17:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1899314716' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi094-78081-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:26.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:26 smithi151 ceph-mon[32923]: from='client.48736 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi094-78081-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[29291]: 159.5 deep-scrub starts 2024-03-12T03:17:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[29291]: 159.5 deep-scrub ok 2024-03-12T03:17:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[29291]: 159.7 scrub starts 2024-03-12T03:17:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[29291]: 159.7 scrub ok 2024-03-12T03:17:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[29291]: 159.1 scrub starts 2024-03-12T03:17:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[29291]: 159.1 scrub ok 2024-03-12T03:17:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[29291]: 159.8 scrub starts 2024-03-12T03:17:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[29291]: 159.8 scrub ok 2024-03-12T03:17:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[29291]: 159.2 scrub starts 2024-03-12T03:17:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[29291]: 159.2 scrub ok 2024-03-12T03:17:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:27.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[29291]: from='client.48736 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi094-78081-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:27.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[29291]: osdmap e136: 8 total, 8 up, 8 in 2024-03-12T03:17:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[40484]: 159.5 deep-scrub starts 2024-03-12T03:17:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[40484]: 159.5 deep-scrub ok 2024-03-12T03:17:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[40484]: 159.7 scrub starts 2024-03-12T03:17:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[40484]: 159.7 scrub ok 2024-03-12T03:17:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[40484]: 159.1 scrub starts 2024-03-12T03:17:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[40484]: 159.1 scrub ok 2024-03-12T03:17:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[40484]: 159.8 scrub starts 2024-03-12T03:17:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[40484]: 159.8 scrub ok 2024-03-12T03:17:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[40484]: 159.2 scrub starts 2024-03-12T03:17:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[40484]: 159.2 scrub ok 2024-03-12T03:17:27.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:27.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[40484]: from='client.48736 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi094-78081-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:27.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:27 smithi094 ceph-mon[40484]: osdmap e136: 8 total, 8 up, 8 in 2024-03-12T03:17:27.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-mon[32923]: 159.5 deep-scrub starts 2024-03-12T03:17:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-mon[32923]: 159.5 deep-scrub ok 2024-03-12T03:17:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-mon[32923]: 159.7 scrub starts 2024-03-12T03:17:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-mon[32923]: 159.7 scrub ok 2024-03-12T03:17:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-mon[32923]: 159.1 scrub starts 2024-03-12T03:17:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-mon[32923]: 159.1 scrub ok 2024-03-12T03:17:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-mon[32923]: 159.8 scrub starts 2024-03-12T03:17:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-mon[32923]: 159.8 scrub ok 2024-03-12T03:17:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-mon[32923]: 159.2 scrub starts 2024-03-12T03:17:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-mon[32923]: 159.2 scrub ok 2024-03-12T03:17:27.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:27.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-mon[32923]: from='client.48736 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi094-78081-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:27.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-mon[32923]: osdmap e136: 8 total, 8 up, 8 in 2024-03-12T03:17:27.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:17:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:17:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[29291]: 159.9 scrub starts 2024-03-12T03:17:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[29291]: 159.9 scrub ok 2024-03-12T03:17:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[29291]: 159.4 scrub starts 2024-03-12T03:17:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[29291]: 159.4 scrub ok 2024-03-12T03:17:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[29291]: pgmap v161: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2024-03-12T03:17:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[29291]: osdmap e137: 8 total, 8 up, 8 in 2024-03-12T03:17:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2966873808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi094-78099-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[29291]: from='client.48739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi094-78099-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[40484]: 159.9 scrub starts 2024-03-12T03:17:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[40484]: 159.9 scrub ok 2024-03-12T03:17:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[40484]: 159.4 scrub starts 2024-03-12T03:17:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[40484]: 159.4 scrub ok 2024-03-12T03:17:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[40484]: pgmap v161: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2024-03-12T03:17:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[40484]: osdmap e137: 8 total, 8 up, 8 in 2024-03-12T03:17:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2966873808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi094-78099-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:28 smithi094 ceph-mon[40484]: from='client.48739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi094-78099-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:28 smithi151 ceph-mon[32923]: 159.9 scrub starts 2024-03-12T03:17:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:28 smithi151 ceph-mon[32923]: 159.9 scrub ok 2024-03-12T03:17:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:28 smithi151 ceph-mon[32923]: 159.4 scrub starts 2024-03-12T03:17:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:28 smithi151 ceph-mon[32923]: 159.4 scrub ok 2024-03-12T03:17:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:28 smithi151 ceph-mon[32923]: pgmap v161: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2024-03-12T03:17:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:28 smithi151 ceph-mon[32923]: osdmap e137: 8 total, 8 up, 8 in 2024-03-12T03:17:28.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2966873808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi094-78099-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:28.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:28 smithi151 ceph-mon[32923]: from='client.48739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi094-78099-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[29291]: 159.6 scrub starts 2024-03-12T03:17:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[29291]: 159.6 scrub ok 2024-03-12T03:17:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[29291]: from='client.48739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi094-78099-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[29291]: osdmap e138: 8 total, 8 up, 8 in 2024-03-12T03:17:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3717078151' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi094-78081-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[29291]: from='client.56552 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi094-78081-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:29.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:29.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[40484]: 159.6 scrub starts 2024-03-12T03:17:29.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[40484]: 159.6 scrub ok 2024-03-12T03:17:29.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:29.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:29.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[40484]: from='client.48739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi094-78099-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:29.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[40484]: osdmap e138: 8 total, 8 up, 8 in 2024-03-12T03:17:29.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:29.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3717078151' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi094-78081-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:29.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[40484]: from='client.56552 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi094-78081-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:29.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:29 smithi151 ceph-mon[32923]: 159.6 scrub starts 2024-03-12T03:17:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:29 smithi151 ceph-mon[32923]: 159.6 scrub ok 2024-03-12T03:17:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:29 smithi151 ceph-mon[32923]: from='client.48739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi094-78099-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:29 smithi151 ceph-mon[32923]: osdmap e138: 8 total, 8 up, 8 in 2024-03-12T03:17:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3717078151' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi094-78081-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:29 smithi151 ceph-mon[32923]: from='client.56552 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi094-78081-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:30.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:17:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:17:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[29291]: pgmap v164: 460 pgs: 32 unknown, 428 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.2 KiB/s wr, 77 op/s 2024-03-12T03:17:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-12T03:17:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[29291]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]': finished 2024-03-12T03:17:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[29291]: from='client.56552 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi094-78081-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[29291]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[29291]: osdmap e139: 8 total, 8 up, 8 in 2024-03-12T03:17:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app2"}]: dispatch 2024-03-12T03:17:30.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:30.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:30.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[29291]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:30.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[40484]: pgmap v164: 460 pgs: 32 unknown, 428 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.2 KiB/s wr, 77 op/s 2024-03-12T03:17:30.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-12T03:17:30.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:30.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[40484]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:30.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]': finished 2024-03-12T03:17:30.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[40484]: from='client.56552 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi094-78081-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:30.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[40484]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:30.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[40484]: osdmap e139: 8 total, 8 up, 8 in 2024-03-12T03:17:30.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app2"}]: dispatch 2024-03-12T03:17:30.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:30.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:30.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:30 smithi094 ceph-mon[40484]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:30 smithi151 ceph-mon[32923]: pgmap v164: 460 pgs: 32 unknown, 428 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.2 KiB/s wr, 77 op/s 2024-03-12T03:17:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-12T03:17:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:30 smithi151 ceph-mon[32923]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]': finished 2024-03-12T03:17:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:30 smithi151 ceph-mon[32923]: from='client.56552 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi094-78081-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:30 smithi151 ceph-mon[32923]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:30 smithi151 ceph-mon[32923]: osdmap e139: 8 total, 8 up, 8 in 2024-03-12T03:17:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app2"}]: dispatch 2024-03-12T03:17:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]: dispatch 2024-03-12T03:17:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:30 smithi151 ceph-mon[32923]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:31.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: Running main() from gmock_main.cc 2024-03-12T03:17:31.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-12T03:17:31.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-12T03:17:31.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-12T03:17:31.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-12T03:17:31.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (4452 ms) 2024-03-12T03:17:31.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-12T03:17:31.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (3511 ms) 2024-03-12T03:17:31.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-12T03:17:31.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (4316 ms) 2024-03-12T03:17:31.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2761 ms) 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (15040 ms total) 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (5036 ms) 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4200 ms) 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (6036 ms) 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (15272 ms total) 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-12T03:17:31.150 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3021 ms) 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1997 ms) 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2006 ms) 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1968 ms) 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8992 ms total) 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4036 ms) 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (3992 ms) 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8028 ms total) 2024-03-12T03:17:31.151 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: 2024-03-12T03:17:31.152 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-12T03:17:31.152 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (74365 ms total) 2024-03-12T03:17:31.152 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-12T03:17:31.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]': finished 2024-03-12T03:17:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[40484]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[40484]: osdmap e140: 8 total, 8 up, 8 in 2024-03-12T03:17:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:17:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2750766340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:17:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[40484]: from='client.48748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[40484]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:17:31.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:31.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]': finished 2024-03-12T03:17:31.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[29291]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:31.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[29291]: osdmap e140: 8 total, 8 up, 8 in 2024-03-12T03:17:31.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:17:31.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2750766340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:31.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:17:31.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[29291]: from='client.48748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:31.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:31 smithi094 ceph-mon[29291]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:17:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3022112134' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi094-78434-15"}]': finished 2024-03-12T03:17:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:31 smithi151 ceph-mon[32923]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi094-78306-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:31.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:31 smithi151 ceph-mon[32923]: osdmap e140: 8 total, 8 up, 8 in 2024-03-12T03:17:31.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:17:31.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2750766340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:31.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:17:31.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:31 smithi151 ceph-mon[32923]: from='client.48748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:31.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:31 smithi151 ceph-mon[32923]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-12T03:17:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[29291]: pgmap v167: 452 pgs: 32 unknown, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-03-12T03:17:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[29291]: from='client.48748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[29291]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-12T03:17:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[29291]: osdmap e141: 8 total, 8 up, 8 in 2024-03-12T03:17:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3980826261' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78081-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1654053169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78440-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:32.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-12T03:17:32.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[29291]: from='client.48757 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78081-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:32.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[29291]: from='client.56570 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78440-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:32.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[29291]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-12T03:17:32.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[40484]: pgmap v167: 452 pgs: 32 unknown, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-03-12T03:17:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[40484]: from='client.48748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[40484]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-12T03:17:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[40484]: osdmap e141: 8 total, 8 up, 8 in 2024-03-12T03:17:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3980826261' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78081-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1654053169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78440-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:32.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-12T03:17:32.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[40484]: from='client.48757 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78081-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:32.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[40484]: from='client.56570 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78440-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:32.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[40484]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-12T03:17:32.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:32 smithi151 ceph-mon[32923]: pgmap v167: 452 pgs: 32 unknown, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-03-12T03:17:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:32 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:32 smithi151 ceph-mon[32923]: from='client.48748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:32 smithi151 ceph-mon[32923]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-12T03:17:32.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:32 smithi151 ceph-mon[32923]: osdmap e141: 8 total, 8 up, 8 in 2024-03-12T03:17:32.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3980826261' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78081-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:32.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1654053169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78440-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:32.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-12T03:17:32.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:32 smithi151 ceph-mon[32923]: from='client.48757 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78081-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:32.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:32 smithi151 ceph-mon[32923]: from='client.56570 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78440-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:32.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:32 smithi151 ceph-mon[32923]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-12T03:17:32.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:34.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[29291]: pgmap v170: 484 pgs: 96 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[29291]: from='client.48757 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78081-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[29291]: from='client.56570 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78440-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[29291]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-12T03:17:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[29291]: osdmap e142: 8 total, 8 up, 8 in 2024-03-12T03:17:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1"}]: dispatch 2024-03-12T03:17:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[29291]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1"}]: dispatch 2024-03-12T03:17:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:17:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:17:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[40484]: pgmap v170: 484 pgs: 96 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[40484]: from='client.48757 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78081-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[40484]: from='client.56570 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78440-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[40484]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-12T03:17:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[40484]: osdmap e142: 8 total, 8 up, 8 in 2024-03-12T03:17:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1"}]: dispatch 2024-03-12T03:17:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[40484]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1"}]: dispatch 2024-03-12T03:17:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:17:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:17:34.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:34 smithi151 ceph-mon[32923]: pgmap v170: 484 pgs: 96 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:34.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:34 smithi151 ceph-mon[32923]: from='client.48757 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78081-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:34.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:34 smithi151 ceph-mon[32923]: from='client.56570 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78440-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:34 smithi151 ceph-mon[32923]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-12T03:17:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:34 smithi151 ceph-mon[32923]: osdmap e142: 8 total, 8 up, 8 in 2024-03-12T03:17:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544131134' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1"}]: dispatch 2024-03-12T03:17:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:34 smithi151 ceph-mon[32923]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1"}]: dispatch 2024-03-12T03:17:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:34 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:17:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:17:35.170 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-12T03:17:35.170 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-12T03:17:35.170 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-12T03:17:35.170 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-12T03:17:35.170 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-12T03:17:35.170 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-12T03:17:35.170 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: 2024-03-12T03:17:35.170 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-12T03:17:35.170 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: seed 78306 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (0 ms) 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (282 ms) 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (7 ms) 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (4 ms) 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (14 ms) 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (4 ms) 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-12T03:17:35.171 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (7 ms) 2024-03-12T03:17:35.172 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-12T03:17:35.172 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (7 ms) 2024-03-12T03:17:35.172 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-12T03:17:35.172 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (4 ms) 2024-03-12T03:17:35.172 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-12T03:17:35.172 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (11 ms) 2024-03-12T03:17:35.172 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-12T03:17:35.172 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (6 ms) 2024-03-12T03:17:35.172 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-12T03:17:35.172 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (8 ms) 2024-03-12T03:17:35.172 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-12T03:17:35.172 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (17 ms) 2024-03-12T03:17:35.172 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-12T03:17:35.173 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-12T03:17:35.173 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-12T03:17:35.173 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1603 ms) 2024-03-12T03:17:35.173 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-12T03:17:35.173 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (350 ms) 2024-03-12T03:17:35.173 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-12T03:17:35.173 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-12T03:17:35.173 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: done waiting, doing copies 2024-03-12T03:17:35.173 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-12T03:17:35.173 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: done waiting 2024-03-12T03:17:35.173 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60814 ms) 2024-03-12T03:17:35.173 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-12T03:17:35.173 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (4 ms) 2024-03-12T03:17:35.174 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-12T03:17:35.174 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (4 ms) 2024-03-12T03:17:35.174 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-12T03:17:35.174 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4528 ms) 2024-03-12T03:17:35.174 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-12T03:17:35.174 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-12T03:17:35.174 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-12T03:17:35.174 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-12T03:17:35.174 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-12T03:17:35.174 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-03-12T03:17:35.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:35 smithi094 ceph-mon[29291]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1"}]': finished 2024-03-12T03:17:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:35 smithi094 ceph-mon[29291]: osdmap e143: 8 total, 8 up, 8 in 2024-03-12T03:17:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2382696252' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi094-78099-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:35 smithi094 ceph-mon[29291]: from='client.56582 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi094-78099-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:35.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:35 smithi094 ceph-mon[40484]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1"}]': finished 2024-03-12T03:17:35.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:35 smithi094 ceph-mon[40484]: osdmap e143: 8 total, 8 up, 8 in 2024-03-12T03:17:35.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2382696252' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi094-78099-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:35.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:35 smithi094 ceph-mon[40484]: from='client.56582 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi094-78099-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:35.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:35.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:35 smithi151 ceph-mon[32923]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi094-78306-1","app":"app1","key":"key1"}]': finished 2024-03-12T03:17:35.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:35 smithi151 ceph-mon[32923]: osdmap e143: 8 total, 8 up, 8 in 2024-03-12T03:17:35.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2382696252' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi094-78099-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:35.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:35 smithi151 ceph-mon[32923]: from='client.56582 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi094-78099-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:35.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:36.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[29291]: pgmap v173: 452 pgs: 32 unknown, 32 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[29291]: from='client.56582 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi094-78099-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[29291]: osdmap e144: 8 total, 8 up, 8 in 2024-03-12T03:17:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2526086238' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi094-78081-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[29291]: from='client.48784 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi094-78081-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78306-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78306-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:36.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[40484]: pgmap v173: 452 pgs: 32 unknown, 32 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:36.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[40484]: from='client.56582 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi094-78099-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:36.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[40484]: osdmap e144: 8 total, 8 up, 8 in 2024-03-12T03:17:36.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2526086238' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi094-78081-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:36.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[40484]: from='client.48784 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi094-78081-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:36.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:36.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:36.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:36.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:36.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78306-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:36.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78306-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:36.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:36 smithi151 ceph-mon[32923]: pgmap v173: 452 pgs: 32 unknown, 32 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:36 smithi151 ceph-mon[32923]: from='client.56582 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi094-78099-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:36 smithi151 ceph-mon[32923]: osdmap e144: 8 total, 8 up, 8 in 2024-03-12T03:17:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2526086238' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi094-78081-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:36.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:36 smithi151 ceph-mon[32923]: from='client.48784 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi094-78081-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:36.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:36.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:36 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:36.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:36.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:36 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:36.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78306-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:36.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:36 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78306-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:36.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[29291]: from='client.48784 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi094-78081-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78306-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[29291]: osdmap e145: 8 total, 8 up, 8 in 2024-03-12T03:17:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78306-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78306-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[29291]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[29291]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:37.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[29291]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:37.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[29291]: from='client.56618 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[40484]: from='client.48784 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi094-78081-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78306-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[40484]: osdmap e145: 8 total, 8 up, 8 in 2024-03-12T03:17:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78306-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78306-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[40484]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:37.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:37.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[40484]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:37.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:37.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[40484]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:37.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:37 smithi094 ceph-mon[40484]: from='client.56618 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-mon[32923]: from='client.48784 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi094-78081-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78306-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-mon[32923]: osdmap e145: 8 total, 8 up, 8 in 2024-03-12T03:17:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78306-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78306-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-mon[32923]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-mon[32923]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-mon[32923]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-mon[32923]: from='client.56618 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:38.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:17:37 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:17:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[29291]: pgmap v176: 388 pgs: 32 unknown, 356 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi094-78440-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[29291]: osdmap e146: 8 total, 8 up, 8 in 2024-03-12T03:17:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3058009079' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi094-78099-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[29291]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi094-78440-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[29291]: from='client.48805 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi094-78099-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78306-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78306-24"}]': finished 2024-03-12T03:17:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[29291]: from='client.48805 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi094-78099-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[29291]: osdmap e147: 8 total, 8 up, 8 in 2024-03-12T03:17:38.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[40484]: pgmap v176: 388 pgs: 32 unknown, 356 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:38.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi094-78440-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:38.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:38.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[40484]: osdmap e146: 8 total, 8 up, 8 in 2024-03-12T03:17:38.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:38.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3058009079' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi094-78099-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:38.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[40484]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi094-78440-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:38.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[40484]: from='client.48805 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi094-78099-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:38.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78306-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78306-24"}]': finished 2024-03-12T03:17:38.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[40484]: from='client.48805 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi094-78099-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:38.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:38.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:38 smithi094 ceph-mon[40484]: osdmap e147: 8 total, 8 up, 8 in 2024-03-12T03:17:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:38 smithi151 ceph-mon[32923]: pgmap v176: 388 pgs: 32 unknown, 356 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-12T03:17:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi094-78440-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:38 smithi151 ceph-mon[32923]: osdmap e146: 8 total, 8 up, 8 in 2024-03-12T03:17:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:38 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3058009079' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi094-78099-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:38 smithi151 ceph-mon[32923]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi094-78440-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:38 smithi151 ceph-mon[32923]: from='client.48805 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi094-78099-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:38 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78306-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78306-24"}]': finished 2024-03-12T03:17:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:38 smithi151 ceph-mon[32923]: from='client.48805 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi094-78099-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:38 smithi151 ceph-mon[32923]: osdmap e147: 8 total, 8 up, 8 in 2024-03-12T03:17:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3457855039' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi094-78081-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[40484]: pgmap v179: 428 pgs: 40 unknown, 32 creating+peering, 356 active+clean; 592 KiB data, 524 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:17:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[40484]: from='client.56624 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi094-78081-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[40484]: from='client.56618 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi094-78440-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]': finished 2024-03-12T03:17:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[40484]: from='client.56624 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi094-78081-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[40484]: osdmap e148: 8 total, 8 up, 8 in 2024-03-12T03:17:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:39.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:39.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:39.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3457855039' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi094-78081-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:39.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[29291]: pgmap v179: 428 pgs: 40 unknown, 32 creating+peering, 356 active+clean; 592 KiB data, 524 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:17:39.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[29291]: from='client.56624 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi094-78081-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:39.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[29291]: from='client.56618 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi094-78440-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]': finished 2024-03-12T03:17:39.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[29291]: from='client.56624 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi094-78081-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:39.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[29291]: osdmap e148: 8 total, 8 up, 8 in 2024-03-12T03:17:39.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:39.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:39.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3457855039' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi094-78081-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:39 smithi151 ceph-mon[32923]: pgmap v179: 428 pgs: 40 unknown, 32 creating+peering, 356 active+clean; 592 KiB data, 524 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:17:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:39 smithi151 ceph-mon[32923]: from='client.56624 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi094-78081-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:39.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:39 smithi151 ceph-mon[32923]: from='client.56618 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi094-78440-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]': finished 2024-03-12T03:17:39.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:39 smithi151 ceph-mon[32923]: from='client.56624 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi094-78081-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:39.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:39 smithi151 ceph-mon[32923]: osdmap e148: 8 total, 8 up, 8 in 2024-03-12T03:17:39.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:39.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:39 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:39.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:17:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:17:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:17:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:41 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:41 smithi094 ceph-mon[29291]: osdmap e149: 8 total, 8 up, 8 in 2024-03-12T03:17:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4230713443' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi094-78099-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:41 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:41.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:41 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:41.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:41 smithi094 ceph-mon[40484]: osdmap e149: 8 total, 8 up, 8 in 2024-03-12T03:17:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4230713443' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi094-78099-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:41 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:41.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:41 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:41.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:41 smithi151 ceph-mon[32923]: osdmap e149: 8 total, 8 up, 8 in 2024-03-12T03:17:41.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:41.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4230713443' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi094-78099-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:41.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:41 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:41.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[29291]: pgmap v182: 436 pgs: 80 unknown, 356 active+clean; 592 KiB data, 524 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:17:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4230713443' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi094-78099-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[29291]: osdmap e150: 8 total, 8 up, 8 in 2024-03-12T03:17:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3681857520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi094-78081-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[29291]: from='client.56636 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi094-78081-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:42.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[40484]: pgmap v182: 436 pgs: 80 unknown, 356 active+clean; 592 KiB data, 524 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:17:42.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4230713443' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi094-78099-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:42.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:42.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[40484]: osdmap e150: 8 total, 8 up, 8 in 2024-03-12T03:17:42.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3681857520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi094-78081-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:42.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[40484]: from='client.56636 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi094-78081-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:42.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:42.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:42 smithi151 ceph-mon[32923]: pgmap v182: 436 pgs: 80 unknown, 356 active+clean; 592 KiB data, 524 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:17:42.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4230713443' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi094-78099-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:42.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:42 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78306-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:42.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:42 smithi151 ceph-mon[32923]: osdmap e150: 8 total, 8 up, 8 in 2024-03-12T03:17:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3681857520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi094-78081-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:42 smithi151 ceph-mon[32923]: from='client.56636 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi094-78081-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:43.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:43 smithi094 ceph-mon[29291]: from='client.56636 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi094-78081-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:43.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:43 smithi094 ceph-mon[29291]: osdmap e151: 8 total, 8 up, 8 in 2024-03-12T03:17:43.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:43.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:43 smithi094 ceph-mon[29291]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:43.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:43 smithi094 ceph-mon[40484]: from='client.56636 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi094-78081-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:43.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:43 smithi094 ceph-mon[40484]: osdmap e151: 8 total, 8 up, 8 in 2024-03-12T03:17:43.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:43.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:43 smithi094 ceph-mon[40484]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:43.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:43 smithi151 ceph-mon[32923]: from='client.56636 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi094-78081-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:43.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:43 smithi151 ceph-mon[32923]: osdmap e151: 8 total, 8 up, 8 in 2024-03-12T03:17:43.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:43.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:43 smithi151 ceph-mon[32923]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:44.162 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP api_snapshots_pp: Running main() from gmock_main.cc 2024-03-12T03:17:44.162 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-12T03:17:44.163 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-12T03:17:44.163 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-12T03:17:44.163 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: seed 78440 2024-03-12T03:17:44.163 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-12T03:17:44.163 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (4455 ms) 2024-03-12T03:17:44.163 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-12T03:17:44.163 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (3501 ms) 2024-03-12T03:17:44.163 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-12T03:17:44.163 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (4324 ms) 2024-03-12T03:17:44.163 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-12T03:17:44.163 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2758 ms) 2024-03-12T03:17:44.163 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-12T03:17:44.164 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (5878 ms) 2024-03-12T03:17:44.164 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (20916 ms total) 2024-03-12T03:17:44.164 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: 2024-03-12T03:17:44.164 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-12T03:17:44.164 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-12T03:17:44.164 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4103 ms) 2024-03-12T03:17:44.164 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-12T03:17:44.164 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4871 ms) 2024-03-12T03:17:44.164 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-12T03:17:44.164 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6947 ms) 2024-03-12T03:17:44.164 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-12T03:17:44.164 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4007 ms) 2024-03-12T03:17:44.165 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-12T03:17:44.165 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2016 ms) 2024-03-12T03:17:44.165 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-12T03:17:44.165 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-g2b0101eb/rpm/el9/BUILD/ceph-19.0.0-1979-g2b0101eb/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-12T03:17:44.165 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: 2024-03-12T03:17:44.165 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-12T03:17:44.165 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-12T03:17:44.165 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi094-78440-7 2024-03-12T03:17:44.165 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-12T03:17:44.165 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17962 ms) 2024-03-12T03:17:44.165 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (39906 ms total) 2024-03-12T03:17:44.165 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: 2024-03-12T03:17:44.166 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-12T03:17:44.166 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-12T03:17:44.166 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (8 ms) 2024-03-12T03:17:44.166 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-12T03:17:44.166 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5066 ms) 2024-03-12T03:17:44.166 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5075 ms total) 2024-03-12T03:17:44.166 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: 2024-03-12T03:17:44.166 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-12T03:17:44.166 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-12T03:17:44.166 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3005 ms) 2024-03-12T03:17:44.166 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-12T03:17:44.169 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnaps snapshots: Running main() from gmock_main.cc 2024-03-12T03:17:44.169 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-03-12T03:17:44.169 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [----------] Global test environment set-up. 2024-03-12T03:17:44.169 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-03-12T03:17:44.169 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-03-12T03:17:44.169 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (12774 ms) 2024-03-12T03:17:44.170 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-03-12T03:17:44.170 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (8597 ms) 2024-03-12T03:17:44.170 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-03-12T03:17:44.170 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4037 ms) 2024-03-12T03:17:44.170 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-03-12T03:17:44.170 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (6278 ms) 2024-03-12T03:17:44.170 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-03-12T03:17:44.170 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7741 ms) 2024-03-12T03:17:44.170 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (39427 ms total) 2024-03-12T03:17:44.170 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: 2024-03-12T03:17:44.170 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-03-12T03:17:44.170 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-03-12T03:17:44.171 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (4667 ms) 2024-03-12T03:17:44.171 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-03-12T03:17:44.171 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (5938 ms) 2024-03-12T03:17:44.171 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-03-12T03:17:44.171 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (7975 ms) 2024-03-12T03:17:44.171 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-03-12T03:17:44.171 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6010 ms) 2024-03-12T03:17:44.171 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-03-12T03:17:44.171 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (3992 ms) 2024-03-12T03:17:44.171 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-03-12T03:17:44.171 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi094-79481-11. 2024-03-12T03:17:44.171 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: Waiting for snaps to purge. 2024-03-12T03:17:44.172 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19036 ms) 2024-03-12T03:17:44.172 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (47618 ms total) 2024-03-12T03:17:44.172 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: 2024-03-12T03:17:44.172 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [----------] Global test environment tear-down 2024-03-12T03:17:44.172 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (87046 ms total) 2024-03-12T03:17:44.172 INFO:tasks.workunit.client.0.smithi094.stdout: snapshots: [ PASSED ] 11 tests. 2024-03-12T03:17:44.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:44 smithi094 ceph-mon[29291]: pgmap v185: 404 pgs: 48 unknown, 356 active+clean; 592 KiB data, 524 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:17:44.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:44 smithi094 ceph-mon[29291]: osdmap e152: 8 total, 8 up, 8 in 2024-03-12T03:17:44.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/879858602' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi094-78099-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:44.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:44.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:44 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:44.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:44.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:44 smithi094 ceph-mon[40484]: pgmap v185: 404 pgs: 48 unknown, 356 active+clean; 592 KiB data, 524 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:17:44.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:44 smithi094 ceph-mon[40484]: osdmap e152: 8 total, 8 up, 8 in 2024-03-12T03:17:44.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/879858602' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi094-78099-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:44.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:44.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:44 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:44.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:44.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:44 smithi151 ceph-mon[32923]: pgmap v185: 404 pgs: 48 unknown, 356 active+clean; 592 KiB data, 524 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:17:44.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:44 smithi151 ceph-mon[32923]: osdmap e152: 8 total, 8 up, 8 in 2024-03-12T03:17:44.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/879858602' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi094-78099-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:44.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:44.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:44 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:44.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/879858602' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi094-78099-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]': finished 2024-03-12T03:17:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[40484]: osdmap e153: 8 total, 8 up, 8 in 2024-03-12T03:17:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2625183514' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi094-78081-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[40484]: from='client.48838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi094-78081-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[40484]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]': finished 2024-03-12T03:17:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[40484]: from='client.48838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi094-78081-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[40484]: osdmap e154: 8 total, 8 up, 8 in 2024-03-12T03:17:45.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/879858602' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi094-78099-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]': finished 2024-03-12T03:17:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[29291]: osdmap e153: 8 total, 8 up, 8 in 2024-03-12T03:17:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2625183514' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi094-78081-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[29291]: from='client.48838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi094-78081-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[29291]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]': finished 2024-03-12T03:17:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[29291]: from='client.48838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi094-78081-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:45 smithi094 ceph-mon[29291]: osdmap e154: 8 total, 8 up, 8 in 2024-03-12T03:17:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/879858602' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi094-78099-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:45 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78306-24"}]': finished 2024-03-12T03:17:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:45 smithi151 ceph-mon[32923]: osdmap e153: 8 total, 8 up, 8 in 2024-03-12T03:17:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2625183514' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi094-78081-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468480639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:45 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]: dispatch 2024-03-12T03:17:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:45 smithi151 ceph-mon[32923]: from='client.48838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi094-78081-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:45 smithi151 ceph-mon[32923]: from='client.56600 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78306-24"}]': finished 2024-03-12T03:17:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:45 smithi151 ceph-mon[32923]: from='client.48838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi094-78081-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:45 smithi151 ceph-mon[32923]: osdmap e154: 8 total, 8 up, 8 in 2024-03-12T03:17:46.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:46 smithi094 ceph-mon[29291]: pgmap v188: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:17:46.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:46.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:46 smithi094 ceph-mon[40484]: pgmap v188: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:17:46.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:46.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:46 smithi151 ceph-mon[32923]: pgmap v188: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:17:46.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:47.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:47 smithi094 ceph-mon[29291]: osdmap e155: 8 total, 8 up, 8 in 2024-03-12T03:17:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/423304727' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi094-78306-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/566798188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78099-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:47 smithi094 ceph-mon[29291]: from='client.48850 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi094-78306-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:47.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:47 smithi094 ceph-mon[40484]: osdmap e155: 8 total, 8 up, 8 in 2024-03-12T03:17:47.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/423304727' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi094-78306-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:47.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/566798188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78099-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:47.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:47 smithi094 ceph-mon[40484]: from='client.48850 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi094-78306-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:47.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:47 smithi151 ceph-mon[32923]: osdmap e155: 8 total, 8 up, 8 in 2024-03-12T03:17:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/423304727' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi094-78306-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/566798188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78099-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:47 smithi151 ceph-mon[32923]: from='client.48850 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi094-78306-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:48.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:17:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:17:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[29291]: pgmap v191: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:17:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/566798188' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78099-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[29291]: from='client.48850 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi094-78306-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[29291]: osdmap e156: 8 total, 8 up, 8 in 2024-03-12T03:17:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1876045718' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi094-78081-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[29291]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1876045718' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi094-78081-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[29291]: osdmap e157: 8 total, 8 up, 8 in 2024-03-12T03:17:48.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[40484]: pgmap v191: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:17:48.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/566798188' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78099-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:48.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[40484]: from='client.48850 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi094-78306-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:48.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[40484]: osdmap e156: 8 total, 8 up, 8 in 2024-03-12T03:17:48.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1876045718' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi094-78081-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:48.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:48.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[40484]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:48.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1876045718' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi094-78081-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:48.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:48 smithi094 ceph-mon[40484]: osdmap e157: 8 total, 8 up, 8 in 2024-03-12T03:17:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:48 smithi151 ceph-mon[32923]: pgmap v191: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:17:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/566798188' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi094-78099-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:48 smithi151 ceph-mon[32923]: from='client.48850 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi094-78306-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:48 smithi151 ceph-mon[32923]: osdmap e156: 8 total, 8 up, 8 in 2024-03-12T03:17:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1876045718' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi094-78081-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:48.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:48 smithi151 ceph-mon[32923]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:48.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1876045718' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi094-78081-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:48.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:48 smithi151 ceph-mon[32923]: osdmap e157: 8 total, 8 up, 8 in 2024-03-12T03:17:49.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:49.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:49.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[29291]: pgmap v194: 364 pgs: 32 creating+peering, 332 active+clean; 592 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:17:49.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:17:49.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[29291]: osdmap e158: 8 total, 8 up, 8 in 2024-03-12T03:17:49.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:49.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/865784997' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi094-78306-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:49.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[29291]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:49.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1351731547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi094-78099-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:49.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[29291]: from='client.48865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi094-78306-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:49.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:49.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:49.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:49.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[40484]: pgmap v194: 364 pgs: 32 creating+peering, 332 active+clean; 592 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:17:49.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:17:49.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[40484]: osdmap e158: 8 total, 8 up, 8 in 2024-03-12T03:17:49.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:49.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/865784997' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi094-78306-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:49.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[40484]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:49.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1351731547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi094-78099-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:49.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[40484]: from='client.48865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi094-78306-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:49.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:49.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:49.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:49.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:49 smithi151 ceph-mon[32923]: pgmap v194: 364 pgs: 32 creating+peering, 332 active+clean; 592 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:17:49.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:49 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:17:49.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:49 smithi151 ceph-mon[32923]: osdmap e158: 8 total, 8 up, 8 in 2024-03-12T03:17:49.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:49.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/865784997' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi094-78306-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:49.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:49 smithi151 ceph-mon[32923]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:49.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1351731547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi094-78099-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:49.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:49 smithi151 ceph-mon[32923]: from='client.48865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi094-78306-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:49.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:50.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:17:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:17:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:17:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[29291]: from='client.56618 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]': finished 2024-03-12T03:17:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1351731547' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi094-78099-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[29291]: from='client.48865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi094-78306-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[29291]: osdmap e159: 8 total, 8 up, 8 in 2024-03-12T03:17:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[29291]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3732296903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi094-78081-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[29291]: from='client.56678 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi094-78081-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:51.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:51.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[40484]: from='client.56618 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]': finished 2024-03-12T03:17:51.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1351731547' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi094-78099-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:51.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[40484]: from='client.48865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi094-78306-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:51.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[40484]: osdmap e159: 8 total, 8 up, 8 in 2024-03-12T03:17:51.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:51.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[40484]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:51.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3732296903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi094-78081-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:51.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[40484]: from='client.56678 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi094-78081-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:51.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:51.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:51 smithi151 ceph-mon[32923]: from='client.56618 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi094-78440-16"}]': finished 2024-03-12T03:17:51.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1351731547' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi094-78099-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:51.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:51 smithi151 ceph-mon[32923]: from='client.48865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi094-78306-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:51 smithi151 ceph-mon[32923]: osdmap e159: 8 total, 8 up, 8 in 2024-03-12T03:17:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1698839484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:51 smithi151 ceph-mon[32923]: from='client.56618 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]: dispatch 2024-03-12T03:17:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3732296903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi094-78081-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:51 smithi151 ceph-mon[32923]: from='client.56678 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi094-78081-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:51.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:52.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[29291]: pgmap v197: 420 pgs: 96 unknown, 324 active+clean; 592 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:17:52.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[29291]: from='client.56618 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]': finished 2024-03-12T03:17:52.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[29291]: from='client.56678 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi094-78081-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:52.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[29291]: osdmap e160: 8 total, 8 up, 8 in 2024-03-12T03:17:52.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:17:52.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:17:52.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:52.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:52.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[40484]: pgmap v197: 420 pgs: 96 unknown, 324 active+clean; 592 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:17:52.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[40484]: from='client.56618 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]': finished 2024-03-12T03:17:52.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[40484]: from='client.56678 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi094-78081-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:52.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[40484]: osdmap e160: 8 total, 8 up, 8 in 2024-03-12T03:17:52.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:17:52.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:17:52.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:52.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:52.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:52 smithi151 ceph-mon[32923]: pgmap v197: 420 pgs: 96 unknown, 324 active+clean; 592 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:17:52.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:52 smithi151 ceph-mon[32923]: from='client.56618 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi094-78440-16"}]': finished 2024-03-12T03:17:52.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:52 smithi151 ceph-mon[32923]: from='client.56678 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi094-78081-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:52.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:52 smithi151 ceph-mon[32923]: osdmap e160: 8 total, 8 up, 8 in 2024-03-12T03:17:52.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:17:52.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:17:52.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:52.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:53.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:53.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: osdmap e161: 8 total, 8 up, 8 in 2024-03-12T03:17:53.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:17:53.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/942667425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi094-78099-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:53.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/823328507' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi094-78306-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:53.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: from='client.48880 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi094-78306-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:53.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi094-78099-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:53.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:53.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:53.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:53.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:53.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi094-78081-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:53.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi094-78081-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:53.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:53.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:53.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:53.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: osdmap e161: 8 total, 8 up, 8 in 2024-03-12T03:17:53.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:17:53.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/942667425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi094-78099-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:53.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/823328507' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi094-78306-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:53.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: from='client.48880 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi094-78306-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:53.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi094-78099-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:53.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:53.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:53.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:53.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:53.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi094-78081-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:53.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi094-78081-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:53.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:53.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:53 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:53.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:53.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: osdmap e161: 8 total, 8 up, 8 in 2024-03-12T03:17:53.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:17:53.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/942667425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi094-78099-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:53.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/823328507' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi094-78306-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:53.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: from='client.48880 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi094-78306-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:53.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi094-78099-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:53.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:53.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:53.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:53.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:53.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi094-78081-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:53.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi094-78081-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:53.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:53.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:53 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:54.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[29291]: pgmap v200: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:17:54.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[29291]: from='client.48880 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi094-78306-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:54.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[29291]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi094-78099-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:54.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[29291]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi094-78081-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:54.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi094-78081-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:54.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:54.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[29291]: osdmap e162: 8 total, 8 up, 8 in 2024-03-12T03:17:54.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[29291]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi094-78081-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[40484]: pgmap v200: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:17:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[40484]: from='client.48880 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi094-78306-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[40484]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi094-78099-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[40484]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi094-78081-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi094-78081-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[40484]: osdmap e162: 8 total, 8 up, 8 in 2024-03-12T03:17:54.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:54 smithi094 ceph-mon[40484]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi094-78081-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:54 smithi151 ceph-mon[32923]: pgmap v200: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:17:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:54 smithi151 ceph-mon[32923]: from='client.48880 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi094-78306-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:54 smithi151 ceph-mon[32923]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi094-78099-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:54 smithi151 ceph-mon[32923]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi094-78081-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi094-78081-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:54 smithi151 ceph-mon[32923]: osdmap e162: 8 total, 8 up, 8 in 2024-03-12T03:17:54.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:54 smithi151 ceph-mon[32923]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi094-78081-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:55.165 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]': finished 2024-03-12T03:17:55.165 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:55.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]': finished 2024-03-12T03:17:55.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:55.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[29291]: osdmap e163: 8 total, 8 up, 8 in 2024-03-12T03:17:55.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:55.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[29291]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:55.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:55.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[29291]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:55.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:55.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[29291]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:55.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]': finished 2024-03-12T03:17:55.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:55.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[40484]: osdmap e163: 8 total, 8 up, 8 in 2024-03-12T03:17:55.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:55.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[40484]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:55.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:55.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[40484]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:55.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:55.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:55 smithi094 ceph-mon[40484]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:55.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:55 smithi151 ceph-mon[32923]: osdmap e163: 8 total, 8 up, 8 in 2024-03-12T03:17:55.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:55.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:55 smithi151 ceph-mon[32923]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:55.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:55.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:55 smithi151 ceph-mon[32923]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:55.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:55.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:55 smithi151 ceph-mon[32923]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:17:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[40484]: pgmap v203: 332 pgs: 8 unknown, 324 active+clean; 592 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:17:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[40484]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi094-78081-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi094-78081-27"}]': finished 2024-03-12T03:17:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[40484]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[40484]: osdmap e164: 8 total, 8 up, 8 in 2024-03-12T03:17:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi094-78306-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[40484]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi094-78306-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2986379974' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi094-78099-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[40484]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi094-78099-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[29291]: pgmap v203: 332 pgs: 8 unknown, 324 active+clean; 592 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:17:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[29291]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi094-78081-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi094-78081-27"}]': finished 2024-03-12T03:17:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[29291]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[29291]: osdmap e164: 8 total, 8 up, 8 in 2024-03-12T03:17:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi094-78306-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[29291]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi094-78306-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2986379974' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi094-78099-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[29291]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi094-78099-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:56 smithi151 ceph-mon[32923]: pgmap v203: 332 pgs: 8 unknown, 324 active+clean; 592 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:17:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:56 smithi151 ceph-mon[32923]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi094-78081-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi094-78081-27"}]': finished 2024-03-12T03:17:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:56 smithi151 ceph-mon[32923]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:17:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:56 smithi151 ceph-mon[32923]: osdmap e164: 8 total, 8 up, 8 in 2024-03-12T03:17:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi094-78306-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:56 smithi151 ceph-mon[32923]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi094-78306-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:17:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2986379974' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi094-78099-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:56 smithi151 ceph-mon[32923]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi094-78099-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:57.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:57 smithi094 ceph-mon[29291]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi094-78099-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:57.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:57.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:57 smithi094 ceph-mon[29291]: osdmap e165: 8 total, 8 up, 8 in 2024-03-12T03:17:57.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:57 smithi094 ceph-mon[40484]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi094-78099-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:57.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:57.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:57 smithi094 ceph-mon[40484]: osdmap e165: 8 total, 8 up, 8 in 2024-03-12T03:17:57.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:57 smithi151 ceph-mon[32923]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi094-78099-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:57.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:57.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:57 smithi151 ceph-mon[32923]: osdmap e165: 8 total, 8 up, 8 in 2024-03-12T03:17:58.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:17:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:17:58.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[29291]: pgmap v206: 372 pgs: 48 unknown, 324 active+clean; 592 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:17:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[29291]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi094-78306-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi094-78306-36"}]': finished 2024-03-12T03:17:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[29291]: osdmap e166: 8 total, 8 up, 8 in 2024-03-12T03:17:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[29291]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[29291]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:58.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[40484]: pgmap v206: 372 pgs: 48 unknown, 324 active+clean; 592 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:17:58.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[40484]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi094-78306-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi094-78306-36"}]': finished 2024-03-12T03:17:58.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[40484]: osdmap e166: 8 total, 8 up, 8 in 2024-03-12T03:17:58.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:58.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[40484]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:58.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:58.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:58 smithi094 ceph-mon[40484]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:58.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:58 smithi151 ceph-mon[32923]: pgmap v206: 372 pgs: 48 unknown, 324 active+clean; 592 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:17:58.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:58 smithi151 ceph-mon[32923]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi094-78306-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi094-78306-36"}]': finished 2024-03-12T03:17:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:58 smithi151 ceph-mon[32923]: osdmap e166: 8 total, 8 up, 8 in 2024-03-12T03:17:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:58 smithi151 ceph-mon[32923]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:58 smithi151 ceph-mon[32923]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:17:59.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: Running main() from gmock_main.cc 2024-03-12T03:17:59.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-12T03:17:59.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [----------] Global test environment set-up. 2024-03-12T03:17:59.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.TooBig (11376 ms) 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (6396 ms) 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (4327 ms) 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3788 ms) 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3473 ms) 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (5189 ms) 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-12T03:17:59.187 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3020 ms) 2024-03-12T03:17:59.188 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-12T03:17:59.188 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3441 ms) 2024-03-12T03:17:59.188 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-12T03:17:59.188 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3516 ms) 2024-03-12T03:17:59.188 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-12T03:17:59.188 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3182 ms) 2024-03-12T03:17:59.188 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-12T03:17:59.188 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2766 ms) 2024-03-12T03:17:59.188 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-12T03:17:59.188 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3016 ms) 2024-03-12T03:17:59.188 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-12T03:17:59.188 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2963 ms) 2024-03-12T03:17:59.188 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3009 ms) 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.Flush (3237 ms) 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2763 ms) 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2999 ms) 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3004 ms) 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3005 ms) 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3017 ms) 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-12T03:17:59.189 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3253 ms) 2024-03-12T03:17:59.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-12T03:17:59.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2726 ms) 2024-03-12T03:17:59.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-12T03:17:59.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3021 ms) 2024-03-12T03:17:59.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-12T03:17:59.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3040 ms) 2024-03-12T03:17:59.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-12T03:17:59.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2937 ms) 2024-03-12T03:17:59.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-12T03:17:59.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3014 ms) 2024-03-12T03:17:59.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [----------] 26 tests from LibRadosAio (95479 ms total) 2024-03-12T03:17:59.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: 2024-03-12T03:17:59.190 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-12T03:17:59.191 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-03-12T03:17:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]': finished 2024-03-12T03:17:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: osdmap e167: 8 total, 8 up, 8 in 2024-03-12T03:17:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1909447587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi094-78099-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:59.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:59.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:59.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:59.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1909447587' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi094-78099-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:59.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]': finished 2024-03-12T03:17:59.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:17:59.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: osdmap e168: 8 total, 8 up, 8 in 2024-03-12T03:17:59.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13"}]: dispatch 2024-03-12T03:17:59.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13"}]: dispatch 2024-03-12T03:17:59.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:59.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]': finished 2024-03-12T03:17:59.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: osdmap e167: 8 total, 8 up, 8 in 2024-03-12T03:17:59.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:59.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1909447587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi094-78099-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:59.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:59.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:59.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:59.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:59.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1909447587' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi094-78099-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:59.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]': finished 2024-03-12T03:17:59.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:17:59.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: osdmap e168: 8 total, 8 up, 8 in 2024-03-12T03:17:59.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13"}]: dispatch 2024-03-12T03:17:59.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:17:59 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13"}]: dispatch 2024-03-12T03:17:59.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:17:59.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi094-78081-27"}]': finished 2024-03-12T03:17:59.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: osdmap e167: 8 total, 8 up, 8 in 2024-03-12T03:17:59.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2648281004' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:59.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1909447587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi094-78099-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:17:59.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]: dispatch 2024-03-12T03:17:59.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:59.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:17:59.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:17:59.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1909447587' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi094-78099-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:17:59.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi094-78081-27"}]': finished 2024-03-12T03:17:59.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:17:59.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: osdmap e168: 8 total, 8 up, 8 in 2024-03-12T03:17:59.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13"}]: dispatch 2024-03-12T03:17:59.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:17:59 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13"}]: dispatch 2024-03-12T03:18:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: pgmap v209: 372 pgs: 32 unknown, 8 creating+peering, 332 active+clean; 604 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-12T03:18:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:18:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:18:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi094-78081-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi094-78081-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:00.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:00.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:00.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13"}]': finished 2024-03-12T03:18:00.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]': finished 2024-03-12T03:18:00.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi094-78081-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:00.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: osdmap e169: 8 total, 8 up, 8 in 2024-03-12T03:18:00.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi094-78081-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:18:00.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi094-78081-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[29291]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:18:00.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: pgmap v209: 372 pgs: 32 unknown, 8 creating+peering, 332 active+clean; 604 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-12T03:18:00.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:18:00.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:18:00.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi094-78081-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:00.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi094-78081-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:00.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:00.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:00.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13"}]': finished 2024-03-12T03:18:00.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]': finished 2024-03-12T03:18:00.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi094-78081-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:00.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: osdmap e169: 8 total, 8 up, 8 in 2024-03-12T03:18:00.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi094-78081-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:18:00.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi094-78081-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-mon[40484]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:18:00.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: pgmap v209: 372 pgs: 32 unknown, 8 creating+peering, 332 active+clean; 604 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-12T03:18:00.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi094-78081-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi094-78081-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-13"}]': finished 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi094-78306-36"}]': finished 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi094-78081-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: osdmap e169: 8 total, 8 up, 8 in 2024-03-12T03:18:00.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi094-78081-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1150027034' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:18:00.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi094-78081-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:00.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:00 smithi151 ceph-mon[32923]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]: dispatch 2024-03-12T03:18:00.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:18:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:18:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:18:01.198 INFO:tasks.workunit.client.0.smithi094.stdout:(67674 ms total) 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (104 ms) 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (104 ms total) 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (37 ms) 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (4 ms) 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (41 ms total) 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: 2024-03-12T03:18:01.199 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (42 ms) 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (2 ms) 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (44 ms total) 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (110 ms) 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (3 ms) 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (113 ms total) 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-12T03:18:01.200 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-12T03:18:01.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1037 ms) 2024-03-12T03:18:01.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1037 ms total) 2024-03-12T03:18:01.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: 2024-03-12T03:18:01.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-12T03:18:01.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (104421 ms total) 2024-03-12T03:18:01.201 INFO:tasks.workunit.client.0.smithi094.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-12T03:18:01.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:01.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:01 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:18:01.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:01 smithi094 ceph-mon[29291]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]': finished 2024-03-12T03:18:01.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:01.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:01 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:18:01.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:01 smithi094 ceph-mon[40484]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]': finished 2024-03-12T03:18:01.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:01.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:01 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:18:01.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:01 smithi151 ceph-mon[32923]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi094-78306-36"}]': finished 2024-03-12T03:18:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: pgmap v212: 332 pgs: 332 active+clean; 604 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-12T03:18:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: osdmap e170: 8 total, 8 up, 8 in 2024-03-12T03:18:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4123655872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi094-78099-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi094-78099-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:18:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:18:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi094-78081-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi094-78081-28"}]': finished 2024-03-12T03:18:02.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi094-78099-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:02.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: osdmap e171: 8 total, 8 up, 8 in 2024-03-12T03:18:02.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:02.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:02.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:02.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]: dispatch 2024-03-12T03:18:02.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[29291]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]: dispatch 2024-03-12T03:18:02.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: pgmap v212: 332 pgs: 332 active+clean; 604 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-12T03:18:02.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:02.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: osdmap e170: 8 total, 8 up, 8 in 2024-03-12T03:18:02.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4123655872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi094-78099-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:02.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi094-78099-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:02.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:02.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:18:02.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:18:02.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi094-78081-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi094-78081-28"}]': finished 2024-03-12T03:18:02.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi094-78099-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:02.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: osdmap e171: 8 total, 8 up, 8 in 2024-03-12T03:18:02.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:02.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:02.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:02.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]: dispatch 2024-03-12T03:18:02.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:02 smithi094 ceph-mon[40484]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]: dispatch 2024-03-12T03:18:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: pgmap v212: 332 pgs: 332 active+clean; 604 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-12T03:18:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: osdmap e170: 8 total, 8 up, 8 in 2024-03-12T03:18:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4123655872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi094-78099-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi094-78099-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:18:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:18:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi094-78081-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi094-78081-28"}]': finished 2024-03-12T03:18:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi094-78099-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: osdmap e171: 8 total, 8 up, 8 in 2024-03-12T03:18:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]: dispatch 2024-03-12T03:18:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:02 smithi151 ceph-mon[32923]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]: dispatch 2024-03-12T03:18:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[40484]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]': finished 2024-03-12T03:18:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[40484]: osdmap e172: 8 total, 8 up, 8 in 2024-03-12T03:18:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-12T03:18:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[40484]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-12T03:18:03.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:03.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:03.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[29291]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]': finished 2024-03-12T03:18:03.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[29291]: osdmap e172: 8 total, 8 up, 8 in 2024-03-12T03:18:03.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:03.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:03.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-12T03:18:03.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:03 smithi094 ceph-mon[29291]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-12T03:18:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:03 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:03 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:03 smithi151 ceph-mon[32923]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]': finished 2024-03-12T03:18:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:03 smithi151 ceph-mon[32923]: osdmap e172: 8 total, 8 up, 8 in 2024-03-12T03:18:03.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:03.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:03.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-12T03:18:03.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:03 smithi151 ceph-mon[32923]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-12T03:18:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[40484]: pgmap v215: 372 pgs: 72 unknown, 300 active+clean; 602 KiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:18:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[40484]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]': finished 2024-03-12T03:18:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[40484]: osdmap e173: 8 total, 8 up, 8 in 2024-03-12T03:18:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2193914045' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi094-78099-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[40484]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[40484]: from='client.56735 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi094-78099-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[29291]: pgmap v215: 372 pgs: 72 unknown, 300 active+clean; 602 KiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:18:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[29291]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]': finished 2024-03-12T03:18:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[29291]: osdmap e173: 8 total, 8 up, 8 in 2024-03-12T03:18:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:04.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:04.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2193914045' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi094-78099-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:04.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[29291]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:04.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[29291]: from='client.56735 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi094-78099-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:04.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:04.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:04 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:04 smithi151 ceph-mon[32923]: pgmap v215: 372 pgs: 72 unknown, 300 active+clean; 602 KiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:04 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:18:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:04 smithi151 ceph-mon[32923]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]': finished 2024-03-12T03:18:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:04 smithi151 ceph-mon[32923]: osdmap e173: 8 total, 8 up, 8 in 2024-03-12T03:18:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2193914045' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi094-78099-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:04 smithi151 ceph-mon[32923]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:04 smithi151 ceph-mon[32923]: from='client.56735 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi094-78099-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:04 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:06.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[40484]: pgmap v218: 364 pgs: 32 unknown, 32 creating+peering, 300 active+clean; 610 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-03-12T03:18:06.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[40484]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]': finished 2024-03-12T03:18:06.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[40484]: from='client.56735 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi094-78099-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:06.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:06.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[40484]: osdmap e174: 8 total, 8 up, 8 in 2024-03-12T03:18:06.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-15"}]: dispatch 2024-03-12T03:18:06.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:06.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:06.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-15"}]: dispatch 2024-03-12T03:18:06.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[40484]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[29291]: pgmap v218: 364 pgs: 32 unknown, 32 creating+peering, 300 active+clean; 610 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-03-12T03:18:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[29291]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]': finished 2024-03-12T03:18:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[29291]: from='client.56735 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi094-78099-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:06.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[29291]: osdmap e174: 8 total, 8 up, 8 in 2024-03-12T03:18:06.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-15"}]: dispatch 2024-03-12T03:18:06.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:06.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:06.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-15"}]: dispatch 2024-03-12T03:18:06.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:06 smithi094 ceph-mon[29291]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:06.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:06 smithi151 ceph-mon[32923]: pgmap v218: 364 pgs: 32 unknown, 32 creating+peering, 300 active+clean; 610 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-03-12T03:18:06.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:06 smithi151 ceph-mon[32923]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi094-78081-28"}]': finished 2024-03-12T03:18:06.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:06 smithi151 ceph-mon[32923]: from='client.56735 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi094-78099-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:06.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:06 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:06.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:06 smithi151 ceph-mon[32923]: osdmap e174: 8 total, 8 up, 8 in 2024-03-12T03:18:06.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-15"}]: dispatch 2024-03-12T03:18:06.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2996656957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:06.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:06.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:06 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-15"}]: dispatch 2024-03-12T03:18:06.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:06 smithi151 ceph-mon[32923]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]: dispatch 2024-03-12T03:18:07.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-15"}]': finished 2024-03-12T03:18:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[29291]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]': finished 2024-03-12T03:18:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[29291]: osdmap e175: 8 total, 8 up, 8 in 2024-03-12T03:18:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-15", "mode": "writeback"}]: dispatch 2024-03-12T03:18:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-15", "mode": "writeback"}]: dispatch 2024-03-12T03:18:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[29291]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[29291]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi094-78081-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:07.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[29291]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi094-78081-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:07.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:07.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-15"}]': finished 2024-03-12T03:18:07.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[40484]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]': finished 2024-03-12T03:18:07.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[40484]: osdmap e175: 8 total, 8 up, 8 in 2024-03-12T03:18:07.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:07.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-15", "mode": "writeback"}]: dispatch 2024-03-12T03:18:07.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-15", "mode": "writeback"}]: dispatch 2024-03-12T03:18:07.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:07.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[40484]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:07.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:07.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[40484]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:07.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi094-78081-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:07.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[40484]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi094-78081-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:07.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:07 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:07.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-15"}]': finished 2024-03-12T03:18:07.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-mon[32923]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi094-78081-28"}]': finished 2024-03-12T03:18:07.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-mon[32923]: osdmap e175: 8 total, 8 up, 8 in 2024-03-12T03:18:07.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:07.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-15", "mode": "writeback"}]: dispatch 2024-03-12T03:18:07.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-15", "mode": "writeback"}]: dispatch 2024-03-12T03:18:07.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:07.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-mon[32923]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:07.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:07.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-mon[32923]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:07.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi094-78081-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:07.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-mon[32923]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi094-78081-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:07.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:08.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:18:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:18:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[40484]: pgmap v221: 332 pgs: 32 creating+peering, 300 active+clean; 610 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 3 op/s 2024-03-12T03:18:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-15", "mode": "writeback"}]': finished 2024-03-12T03:18:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[40484]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi094-78081-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[40484]: osdmap e176: 8 total, 8 up, 8 in 2024-03-12T03:18:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3907080750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi094-78099-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi094-78081-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:18:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[40484]: from='client.56744 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi094-78099-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[40484]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi094-78081-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:08.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]': finished 2024-03-12T03:18:08.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[29291]: pgmap v221: 332 pgs: 32 creating+peering, 300 active+clean; 610 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 3 op/s 2024-03-12T03:18:08.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-15", "mode": "writeback"}]': finished 2024-03-12T03:18:08.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[29291]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi094-78081-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:08.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[29291]: osdmap e176: 8 total, 8 up, 8 in 2024-03-12T03:18:08.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3907080750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi094-78099-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:08.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:08.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi094-78081-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:08.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:18:08.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[29291]: from='client.56744 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi094-78099-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:08.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[29291]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi094-78081-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:08.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]': finished 2024-03-12T03:18:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:08 smithi151 ceph-mon[32923]: pgmap v221: 332 pgs: 32 creating+peering, 300 active+clean; 610 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 3 op/s 2024-03-12T03:18:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:08 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-15", "mode": "writeback"}]': finished 2024-03-12T03:18:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:08 smithi151 ceph-mon[32923]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi094-78081-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:08 smithi151 ceph-mon[32923]: osdmap e176: 8 total, 8 up, 8 in 2024-03-12T03:18:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3907080750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi094-78099-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi094-78081-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:18:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:08 smithi151 ceph-mon[32923]: from='client.56744 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi094-78099-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:08 smithi151 ceph-mon[32923]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi094-78081-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]': finished 2024-03-12T03:18:09.143 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-03-12T03:18:09.144 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-12T03:18:09.144 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-12T03:18:09.144 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-12T03:18:09.144 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-12T03:18:09.144 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (11372 ms) 2024-03-12T03:18:09.144 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-12T03:18:09.144 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (21968 ms) 2024-03-12T03:18:09.144 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-12T03:18:09.144 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6557 ms) 2024-03-12T03:18:09.144 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-12T03:18:09.144 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2518 ms) 2024-03-12T03:18:09.144 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3336 ms) 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3718 ms) 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3010 ms) 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3004 ms) 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2968 ms) 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3000 ms) 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3006 ms) 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-12T03:18:09.145 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2995 ms) 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3007 ms) 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3008 ms) 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3009 ms) 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3016 ms) 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2969 ms) 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3006 ms) 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-12T03:18:09.146 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3020 ms) 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2972 ms) 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3019 ms) 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3254 ms) 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2775 ms) 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3008 ms) 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2953 ms) 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-12T03:18:09.147 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3002 ms) 2024-03-12T03:18:09.148 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-12T03:18:09.148 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAihotsECPP.SnapRemovePP (2020 ms) 2024-03-12T03:18:09.148 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-12T03:18:09.148 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2030 ms) 2024-03-12T03:18:09.148 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-12T03:18:09.148 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1940 ms) 2024-03-12T03:18:09.148 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8995 ms total) 2024-03-12T03:18:09.148 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: 2024-03-12T03:18:09.148 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-12T03:18:09.148 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-12T03:18:09.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (3805 ms) 2024-03-12T03:18:09.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-12T03:18:09.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3986 ms) 2024-03-12T03:18:09.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-12T03:18:09.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3945 ms) 2024-03-12T03:18:09.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11736 ms total) 2024-03-12T03:18:09.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: 2024-03-12T03:18:09.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-12T03:18:09.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (112357 ms total) 2024-03-12T03:18:09.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-12T03:18:09.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-12T03:18:09.149 INFO:tasks.workunit.client.0.smithi094.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-12T03:18:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: from='client.56744 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi094-78099-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: osdmap e177: 8 total, 8 up, 8 in 2024-03-12T03:18:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:18:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:09.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:09.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi094-78081-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi094-78081-29"}]': finished 2024-03-12T03:18:09.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]': finished 2024-03-12T03:18:09.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:09.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: osdmap e178: 8 total, 8 up, 8 in 2024-03-12T03:18:09.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15"}]: dispatch 2024-03-12T03:18:09.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15"}]: dispatch 2024-03-12T03:18:09.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:09.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:09.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: from='client.56744 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi094-78099-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:09.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:09.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: osdmap e177: 8 total, 8 up, 8 in 2024-03-12T03:18:09.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:18:09.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:09.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:09.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi094-78081-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi094-78081-29"}]': finished 2024-03-12T03:18:09.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]': finished 2024-03-12T03:18:09.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:09.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: osdmap e178: 8 total, 8 up, 8 in 2024-03-12T03:18:09.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15"}]: dispatch 2024-03-12T03:18:09.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15"}]: dispatch 2024-03-12T03:18:09.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: from='client.56744 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi094-78099-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: osdmap e177: 8 total, 8 up, 8 in 2024-03-12T03:18:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]: dispatch 2024-03-12T03:18:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi094-78081-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi094-78081-29"}]': finished 2024-03-12T03:18:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/114341178' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi094-78440-21"}]': finished 2024-03-12T03:18:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: osdmap e178: 8 total, 8 up, 8 in 2024-03-12T03:18:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15"}]: dispatch 2024-03-12T03:18:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15"}]: dispatch 2024-03-12T03:18:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:10.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:18:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:18:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[29291]: pgmap v224: 356 pgs: 32 creating+peering, 324 active+clean; 586 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15"}]': finished 2024-03-12T03:18:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[29291]: osdmap e179: 8 total, 8 up, 8 in 2024-03-12T03:18:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3221712127' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi094-78099-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[29291]: from='client.56753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi094-78099-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:10.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[40484]: pgmap v224: 356 pgs: 32 creating+peering, 324 active+clean; 586 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:10.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:10.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:10.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15"}]': finished 2024-03-12T03:18:10.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[40484]: osdmap e179: 8 total, 8 up, 8 in 2024-03-12T03:18:10.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3221712127' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi094-78099-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:10.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[40484]: from='client.56753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi094-78099-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:10.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:10 smithi151 ceph-mon[32923]: pgmap v224: 356 pgs: 32 creating+peering, 324 active+clean; 586 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:10 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:10 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:10 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-15"}]': finished 2024-03-12T03:18:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:10 smithi151 ceph-mon[32923]: osdmap e179: 8 total, 8 up, 8 in 2024-03-12T03:18:10.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3221712127' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi094-78099-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:10.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:10 smithi151 ceph-mon[32923]: from='client.56753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi094-78099-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:10.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:12 smithi094 ceph-mon[29291]: pgmap v227: 364 pgs: 40 unknown, 324 active+clean; 586 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:12 smithi094 ceph-mon[29291]: from='client.56753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi094-78099-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:12 smithi094 ceph-mon[29291]: osdmap e180: 8 total, 8 up, 8 in 2024-03-12T03:18:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:12 smithi094 ceph-mon[29291]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:12.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:12 smithi094 ceph-mon[40484]: pgmap v227: 364 pgs: 40 unknown, 324 active+clean; 586 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:12.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:12 smithi094 ceph-mon[40484]: from='client.56753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi094-78099-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:12.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:12 smithi094 ceph-mon[40484]: osdmap e180: 8 total, 8 up, 8 in 2024-03-12T03:18:12.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:12.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:12 smithi094 ceph-mon[40484]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:12.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:12.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:12 smithi151 ceph-mon[32923]: pgmap v227: 364 pgs: 40 unknown, 324 active+clean; 586 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:12.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:12 smithi151 ceph-mon[32923]: from='client.56753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi094-78099-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:12.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:12 smithi151 ceph-mon[32923]: osdmap e180: 8 total, 8 up, 8 in 2024-03-12T03:18:12.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:12.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:12 smithi151 ceph-mon[32923]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:12.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[40484]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]': finished 2024-03-12T03:18:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[40484]: osdmap e181: 8 total, 8 up, 8 in 2024-03-12T03:18:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[40484]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[40484]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]': finished 2024-03-12T03:18:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[40484]: osdmap e182: 8 total, 8 up, 8 in 2024-03-12T03:18:13.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3134408602' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi094-78099-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:13.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[29291]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]': finished 2024-03-12T03:18:13.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[29291]: osdmap e181: 8 total, 8 up, 8 in 2024-03-12T03:18:13.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:13.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[29291]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:13.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:13.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:13.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:13.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[29291]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]': finished 2024-03-12T03:18:13.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:13.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[29291]: osdmap e182: 8 total, 8 up, 8 in 2024-03-12T03:18:13.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3134408602' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi094-78099-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:13.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:13 smithi151 ceph-mon[32923]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi094-78081-29"}]': finished 2024-03-12T03:18:13.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:13 smithi151 ceph-mon[32923]: osdmap e181: 8 total, 8 up, 8 in 2024-03-12T03:18:13.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3437219768' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:13.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:13 smithi151 ceph-mon[32923]: from='client.48931 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]: dispatch 2024-03-12T03:18:13.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:13.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:13.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:13 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:13.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:13 smithi151 ceph-mon[32923]: from='client.48931 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi094-78081-29"}]': finished 2024-03-12T03:18:13.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:13 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:13.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:13 smithi151 ceph-mon[32923]: osdmap e182: 8 total, 8 up, 8 in 2024-03-12T03:18:13.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3134408602' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi094-78099-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: pgmap v230: 324 pgs: 32 unknown, 292 active+clean; 586 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.56756 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi094-78099-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi094-78081-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:14.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi094-78081-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:14.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.56756 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi094-78099-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:14.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:14.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.48949 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi094-78081-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:14.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: osdmap e183: 8 total, 8 up, 8 in 2024-03-12T03:18:14.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-17"}]: dispatch 2024-03-12T03:18:14.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi094-78081-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-17"}]: dispatch 2024-03-12T03:18:14.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi094-78081-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:14.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: pgmap v230: 324 pgs: 32 unknown, 292 active+clean; 586 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:14.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.56756 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi094-78099-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:14.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi094-78081-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi094-78081-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.56756 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi094-78099-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.48949 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi094-78081-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: osdmap e183: 8 total, 8 up, 8 in 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-17"}]: dispatch 2024-03-12T03:18:14.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi094-78081-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-17"}]: dispatch 2024-03-12T03:18:14.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi094-78081-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:14.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: pgmap v230: 324 pgs: 32 unknown, 292 active+clean; 586 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.56756 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi094-78099-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi094-78081-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi094-78081-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.56756 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi094-78099-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.48949 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi094-78081-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: osdmap e183: 8 total, 8 up, 8 in 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-17"}]: dispatch 2024-03-12T03:18:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi094-78081-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-17"}]: dispatch 2024-03-12T03:18:14.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi094-78081-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:14.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[29291]: pgmap v233: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:18:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-17"}]': finished 2024-03-12T03:18:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-17", "mode": "writeback"}]: dispatch 2024-03-12T03:18:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[29291]: osdmap e184: 8 total, 8 up, 8 in 2024-03-12T03:18:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:16.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-17", "mode": "writeback"}]: dispatch 2024-03-12T03:18:16.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[40484]: pgmap v233: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:18:16.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-17"}]': finished 2024-03-12T03:18:16.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-17", "mode": "writeback"}]: dispatch 2024-03-12T03:18:16.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:16.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[40484]: osdmap e184: 8 total, 8 up, 8 in 2024-03-12T03:18:16.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:16.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:16 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-17", "mode": "writeback"}]: dispatch 2024-03-12T03:18:16.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:16 smithi151 ceph-mon[32923]: pgmap v233: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:18:16.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:16 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-17"}]': finished 2024-03-12T03:18:16.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-17", "mode": "writeback"}]: dispatch 2024-03-12T03:18:16.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:16.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:16 smithi151 ceph-mon[32923]: osdmap e184: 8 total, 8 up, 8 in 2024-03-12T03:18:16.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:16 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:16.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:16 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-17", "mode": "writeback"}]: dispatch 2024-03-12T03:18:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[29291]: from='client.48949 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi094-78081-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi094-78081-30"}]': finished 2024-03-12T03:18:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-17", "mode": "writeback"}]': finished 2024-03-12T03:18:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[29291]: osdmap e185: 8 total, 8 up, 8 in 2024-03-12T03:18:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2924902146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi094-78099-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:17.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[40484]: from='client.48949 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi094-78081-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi094-78081-30"}]': finished 2024-03-12T03:18:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-17", "mode": "writeback"}]': finished 2024-03-12T03:18:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[40484]: osdmap e185: 8 total, 8 up, 8 in 2024-03-12T03:18:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2924902146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi094-78099-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:17 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:17 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:17 smithi151 ceph-mon[32923]: from='client.48949 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi094-78081-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi094-78081-30"}]': finished 2024-03-12T03:18:17.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:17 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-17", "mode": "writeback"}]': finished 2024-03-12T03:18:17.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:17 smithi151 ceph-mon[32923]: osdmap e185: 8 total, 8 up, 8 in 2024-03-12T03:18:17.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:17.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2924902146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi094-78099-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:17.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:17.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:17 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:18.163 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:18:17 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:18:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[29291]: pgmap v236: 364 pgs: 40 unknown, 32 creating+peering, 292 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:18:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2924902146' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi094-78099-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[29291]: osdmap e186: 8 total, 8 up, 8 in 2024-03-12T03:18:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17"}]: dispatch 2024-03-12T03:18:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17"}]: dispatch 2024-03-12T03:18:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17"}]': finished 2024-03-12T03:18:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[29291]: osdmap e187: 8 total, 8 up, 8 in 2024-03-12T03:18:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:18.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[29291]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[40484]: pgmap v236: 364 pgs: 40 unknown, 32 creating+peering, 292 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:18:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2924902146' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi094-78099-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[40484]: osdmap e186: 8 total, 8 up, 8 in 2024-03-12T03:18:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17"}]: dispatch 2024-03-12T03:18:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17"}]: dispatch 2024-03-12T03:18:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17"}]': finished 2024-03-12T03:18:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[40484]: osdmap e187: 8 total, 8 up, 8 in 2024-03-12T03:18:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:18.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:18 smithi094 ceph-mon[40484]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:18.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:18 smithi151 ceph-mon[32923]: pgmap v236: 364 pgs: 40 unknown, 32 creating+peering, 292 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:18:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2924902146' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi094-78099-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:18 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:18 smithi151 ceph-mon[32923]: osdmap e186: 8 total, 8 up, 8 in 2024-03-12T03:18:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17"}]: dispatch 2024-03-12T03:18:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:18 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17"}]: dispatch 2024-03-12T03:18:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:18 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:18 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-17"}]': finished 2024-03-12T03:18:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:18 smithi151 ceph-mon[32923]: osdmap e187: 8 total, 8 up, 8 in 2024-03-12T03:18:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:18 smithi151 ceph-mon[32923]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-12T03:18:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-12T03:18:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[29291]: from='client.48949 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]': finished 2024-03-12T03:18:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-12T03:18:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[29291]: osdmap e188: 8 total, 8 up, 8 in 2024-03-12T03:18:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:19.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2648266258' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi094-78099-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:19.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[29291]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:19.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[29291]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi094-78099-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-12T03:18:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-12T03:18:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[40484]: from='client.48949 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]': finished 2024-03-12T03:18:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-12T03:18:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[40484]: osdmap e188: 8 total, 8 up, 8 in 2024-03-12T03:18:19.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:19.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2648266258' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi094-78099-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:19.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[40484]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:19.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:19 smithi094 ceph-mon[40484]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi094-78099-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-12T03:18:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:19 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-12T03:18:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:19 smithi151 ceph-mon[32923]: from='client.48949 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi094-78081-30"}]': finished 2024-03-12T03:18:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:19 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-12T03:18:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:19 smithi151 ceph-mon[32923]: osdmap e188: 8 total, 8 up, 8 in 2024-03-12T03:18:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1407370179' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2648266258' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi094-78099-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:19 smithi151 ceph-mon[32923]: from='client.48949 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]: dispatch 2024-03-12T03:18:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:19 smithi151 ceph-mon[32923]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi094-78099-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:20.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:18:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:18:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: pgmap v239: 324 pgs: 324 active+clean; 586 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-12T03:18:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.48949 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]': finished 2024-03-12T03:18:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi094-78099-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: osdmap e189: 8 total, 8 up, 8 in 2024-03-12T03:18:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi094-78081-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi094-78081-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]: dispatch 2024-03-12T03:18:20.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[29291]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]: dispatch 2024-03-12T03:18:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: pgmap v239: 324 pgs: 324 active+clean; 586 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-12T03:18:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.48949 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]': finished 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi094-78099-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: osdmap e189: 8 total, 8 up, 8 in 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi094-78081-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi094-78081-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]: dispatch 2024-03-12T03:18:20.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:20 smithi094 ceph-mon[40484]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]: dispatch 2024-03-12T03:18:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: pgmap v239: 324 pgs: 324 active+clean; 586 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-12T03:18:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.48949 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi094-78081-30"}]': finished 2024-03-12T03:18:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi094-78099-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: osdmap e189: 8 total, 8 up, 8 in 2024-03-12T03:18:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi094-78081-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi094-78081-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]: dispatch 2024-03-12T03:18:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:20 smithi151 ceph-mon[32923]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]: dispatch 2024-03-12T03:18:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2648266258' entity='client.admin' cmd=[{ 2024-03-12T03:18:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: "prefix": "osd pool set", 2024-03-12T03:18:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: "pool": "PoolEIOFlag_smithi094-78099-33", 2024-03-12T03:18:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: "var": "eio", 2024-03-12T03:18:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: "val": "true" 2024-03-12T03:18:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: }]: dispatch 2024-03-12T03:18:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.48961 ' entity='client.admin' cmd=[{ 2024-03-12T03:18:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: "prefix": "osd pool set", 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: "pool": "PoolEIOFlag_smithi094-78099-33", 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: "var": "eio", 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: "val": "true" 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: }]: dispatch 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.48967 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi094-78081-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]': finished 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.48961 ' entity='client.admin' cmd='[{ 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: "prefix": "osd pool set", 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: "pool": "PoolEIOFlag_smithi094-78099-33", 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: "var": "eio", 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: "val": "true" 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: }]': finished 2024-03-12T03:18:21.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi094-78081-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:21.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: osdmap e190: 8 total, 8 up, 8 in 2024-03-12T03:18:21.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:21.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:21.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-12T03:18:21.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:21.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi094-78081-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:21.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-12T03:18:21.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:21.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2648266258' entity='client.admin' cmd=[{ 2024-03-12T03:18:21.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: "prefix": "osd pool set", 2024-03-12T03:18:21.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: "pool": "PoolEIOFlag_smithi094-78099-33", 2024-03-12T03:18:21.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: "var": "eio", 2024-03-12T03:18:21.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: "val": "true" 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: }]: dispatch 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.48961 ' entity='client.admin' cmd=[{ 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: "prefix": "osd pool set", 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: "pool": "PoolEIOFlag_smithi094-78099-33", 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: "var": "eio", 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: "val": "true" 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: }]: dispatch 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.48967 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi094-78081-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]': finished 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.48961 ' entity='client.admin' cmd='[{ 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: "prefix": "osd pool set", 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: "pool": "PoolEIOFlag_smithi094-78099-33", 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: "var": "eio", 2024-03-12T03:18:21.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: "val": "true" 2024-03-12T03:18:21.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: }]': finished 2024-03-12T03:18:21.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi094-78081-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:21.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: osdmap e190: 8 total, 8 up, 8 in 2024-03-12T03:18:21.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:21.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:21.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-12T03:18:21.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:21.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi094-78081-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:21.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-12T03:18:21.714 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:21 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:21.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2648266258' entity='client.admin' cmd=[{ 2024-03-12T03:18:21.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: "prefix": "osd pool set", 2024-03-12T03:18:21.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: "pool": "PoolEIOFlag_smithi094-78099-33", 2024-03-12T03:18:21.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: "var": "eio", 2024-03-12T03:18:21.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: "val": "true" 2024-03-12T03:18:21.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: }]: dispatch 2024-03-12T03:18:21.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.48961 ' entity='client.admin' cmd=[{ 2024-03-12T03:18:21.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: "prefix": "osd pool set", 2024-03-12T03:18:21.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: "pool": "PoolEIOFlag_smithi094-78099-33", 2024-03-12T03:18:21.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: "var": "eio", 2024-03-12T03:18:21.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: "val": "true" 2024-03-12T03:18:21.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: }]: dispatch 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.48967 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi094-78081-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pg_num","val":"11"}]': finished 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.48961 ' entity='client.admin' cmd='[{ 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: "prefix": "osd pool set", 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: "pool": "PoolEIOFlag_smithi094-78099-33", 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: "var": "eio", 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: "val": "true" 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: }]': finished 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi094-78081-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: osdmap e190: 8 total, 8 up, 8 in 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-12T03:18:21.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:21.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi094-78081-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:21.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-12T03:18:21.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:21 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[29291]: pgmap v242: 356 pgs: 64 unknown, 292 active+clean; 586 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:18:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[29291]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]': finished 2024-03-12T03:18:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:22.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-19"}]: dispatch 2024-03-12T03:18:22.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[29291]: osdmap e191: 8 total, 8 up, 8 in 2024-03-12T03:18:22.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-19"}]: dispatch 2024-03-12T03:18:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[40484]: pgmap v242: 356 pgs: 64 unknown, 292 active+clean; 586 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:18:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[40484]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]': finished 2024-03-12T03:18:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-19"}]: dispatch 2024-03-12T03:18:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[40484]: osdmap e191: 8 total, 8 up, 8 in 2024-03-12T03:18:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:22 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-19"}]: dispatch 2024-03-12T03:18:22.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:22 smithi151 ceph-mon[32923]: pgmap v242: 356 pgs: 64 unknown, 292 active+clean; 586 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:18:22.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:22 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:22.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:22 smithi151 ceph-mon[32923]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi094-78177-1","var":"pgp_num","val":"11"}]': finished 2024-03-12T03:18:22.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:22 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:22.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:22.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-19"}]: dispatch 2024-03-12T03:18:22.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:22 smithi151 ceph-mon[32923]: osdmap e191: 8 total, 8 up, 8 in 2024-03-12T03:18:22.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:22 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-19"}]: dispatch 2024-03-12T03:18:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[29291]: pgmap v245: 324 pgs: 32 unknown, 292 active+clean; 586 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[29291]: from='client.48967 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi094-78081-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi094-78081-31"}]': finished 2024-03-12T03:18:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-19"}]': finished 2024-03-12T03:18:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[29291]: osdmap e192: 8 total, 8 up, 8 in 2024-03-12T03:18:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3752574764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi094-78099-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-19", "mode": "writeback"}]: dispatch 2024-03-12T03:18:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[29291]: from='client.56783 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi094-78099-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-19", "mode": "writeback"}]: dispatch 2024-03-12T03:18:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[40484]: pgmap v245: 324 pgs: 32 unknown, 292 active+clean; 586 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[40484]: from='client.48967 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi094-78081-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi094-78081-31"}]': finished 2024-03-12T03:18:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-19"}]': finished 2024-03-12T03:18:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[40484]: osdmap e192: 8 total, 8 up, 8 in 2024-03-12T03:18:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3752574764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi094-78099-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-19", "mode": "writeback"}]: dispatch 2024-03-12T03:18:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[40484]: from='client.56783 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi094-78099-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:24 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-19", "mode": "writeback"}]: dispatch 2024-03-12T03:18:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:24 smithi151 ceph-mon[32923]: pgmap v245: 324 pgs: 32 unknown, 292 active+clean; 586 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:24 smithi151 ceph-mon[32923]: from='client.48967 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi094-78081-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi094-78081-31"}]': finished 2024-03-12T03:18:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:24 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-19"}]': finished 2024-03-12T03:18:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:24 smithi151 ceph-mon[32923]: osdmap e192: 8 total, 8 up, 8 in 2024-03-12T03:18:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:24 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:24.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:24 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3752574764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi094-78099-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:24.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:24 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-19", "mode": "writeback"}]: dispatch 2024-03-12T03:18:24.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:24 smithi151 ceph-mon[32923]: from='client.56783 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi094-78099-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:24.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:24 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-19", "mode": "writeback"}]: dispatch 2024-03-12T03:18:25.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[29291]: from='client.56783 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi094-78099-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-19", "mode": "writeback"}]': finished 2024-03-12T03:18:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[29291]: osdmap e193: 8 total, 8 up, 8 in 2024-03-12T03:18:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-12T03:18:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-12T03:18:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:25.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:25.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[40484]: from='client.56783 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi094-78099-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:25.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-19", "mode": "writeback"}]': finished 2024-03-12T03:18:25.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[40484]: osdmap e193: 8 total, 8 up, 8 in 2024-03-12T03:18:25.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:25.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:25.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-12T03:18:25.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:25 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-12T03:18:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:25 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:25 smithi151 ceph-mon[32923]: from='client.56783 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi094-78099-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:25 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-19", "mode": "writeback"}]': finished 2024-03-12T03:18:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:25 smithi151 ceph-mon[32923]: osdmap e193: 8 total, 8 up, 8 in 2024-03-12T03:18:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:25 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:25 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-12T03:18:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:25 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-12T03:18:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[29291]: pgmap v248: 364 pgs: 40 unknown, 324 active+clean; 586 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-12T03:18:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-12T03:18:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19"}]: dispatch 2024-03-12T03:18:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[29291]: osdmap e194: 8 total, 8 up, 8 in 2024-03-12T03:18:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19"}]: dispatch 2024-03-12T03:18:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[29291]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:26.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[40484]: pgmap v248: 364 pgs: 40 unknown, 324 active+clean; 586 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-12T03:18:26.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:26.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:26.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-12T03:18:26.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19"}]: dispatch 2024-03-12T03:18:26.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[40484]: osdmap e194: 8 total, 8 up, 8 in 2024-03-12T03:18:26.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19"}]: dispatch 2024-03-12T03:18:26.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:26.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:26 smithi094 ceph-mon[40484]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:26 smithi151 ceph-mon[32923]: pgmap v248: 364 pgs: 40 unknown, 324 active+clean; 586 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-12T03:18:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:26 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:26 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-12T03:18:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19"}]: dispatch 2024-03-12T03:18:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:26 smithi151 ceph-mon[32923]: osdmap e194: 8 total, 8 up, 8 in 2024-03-12T03:18:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:26 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19"}]: dispatch 2024-03-12T03:18:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:26 smithi151 ceph-mon[32923]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:27.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19"}]': finished 2024-03-12T03:18:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[29291]: from='client.48967 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]': finished 2024-03-12T03:18:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[29291]: osdmap e195: 8 total, 8 up, 8 in 2024-03-12T03:18:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1975866651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi094-78099-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[29291]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[29291]: from='client.56789 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi094-78099-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:27.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:27.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19"}]': finished 2024-03-12T03:18:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[40484]: from='client.48967 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]': finished 2024-03-12T03:18:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[40484]: osdmap e195: 8 total, 8 up, 8 in 2024-03-12T03:18:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1975866651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi094-78099-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[40484]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:27 smithi094 ceph-mon[40484]: from='client.56789 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi094-78099-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:27.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:27.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:27 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:27.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:27 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-19"}]': finished 2024-03-12T03:18:27.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:27 smithi151 ceph-mon[32923]: from='client.48967 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi094-78081-31"}]': finished 2024-03-12T03:18:27.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:27 smithi151 ceph-mon[32923]: osdmap e195: 8 total, 8 up, 8 in 2024-03-12T03:18:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/860894324' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1975866651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi094-78099-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:27 smithi151 ceph-mon[32923]: from='client.48967 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]: dispatch 2024-03-12T03:18:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:27 smithi151 ceph-mon[32923]: from='client.56789 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi094-78099-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:28.160 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:18:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:18:28.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[29291]: pgmap v251: 356 pgs: 32 unknown, 324 active+clean; 586 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-12T03:18:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[29291]: from='client.48967 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]': finished 2024-03-12T03:18:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[29291]: from='client.56789 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi094-78099-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[29291]: osdmap e196: 8 total, 8 up, 8 in 2024-03-12T03:18:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[29291]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[29291]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi094-78081-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[29291]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi094-78081-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:28.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[40484]: pgmap v251: 356 pgs: 32 unknown, 324 active+clean; 586 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-12T03:18:28.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:28.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[40484]: from='client.48967 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]': finished 2024-03-12T03:18:28.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[40484]: from='client.56789 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi094-78099-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:28.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[40484]: osdmap e196: 8 total, 8 up, 8 in 2024-03-12T03:18:28.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:28.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[40484]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:28.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:28.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[40484]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:28.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi094-78081-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:28.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:28 smithi094 ceph-mon[40484]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi094-78081-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:28.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:28 smithi151 ceph-mon[32923]: pgmap v251: 356 pgs: 32 unknown, 324 active+clean; 586 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-12T03:18:28.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:28.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:28 smithi151 ceph-mon[32923]: from='client.48967 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi094-78081-31"}]': finished 2024-03-12T03:18:28.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:28 smithi151 ceph-mon[32923]: from='client.56789 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi094-78099-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:28.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:28 smithi151 ceph-mon[32923]: osdmap e196: 8 total, 8 up, 8 in 2024-03-12T03:18:28.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:28.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:28 smithi151 ceph-mon[32923]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:28.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:28.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:28 smithi151 ceph-mon[32923]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:28.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi094-78081-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:28.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:28 smithi151 ceph-mon[32923]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi094-78081-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:29.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:29.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:29.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[29291]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi094-78081-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:29.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[29291]: osdmap e197: 8 total, 8 up, 8 in 2024-03-12T03:18:29.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi094-78081-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:29.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[29291]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi094-78081-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:29.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:29.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:29.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:29.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:29.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[40484]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi094-78081-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:29.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[40484]: osdmap e197: 8 total, 8 up, 8 in 2024-03-12T03:18:29.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi094-78081-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:29.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[40484]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi094-78081-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:29.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:29.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:29 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:29.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:29.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:29.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:29 smithi151 ceph-mon[32923]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi094-78081-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:29.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:29 smithi151 ceph-mon[32923]: osdmap e197: 8 total, 8 up, 8 in 2024-03-12T03:18:29.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi094-78081-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:29.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:29 smithi151 ceph-mon[32923]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi094-78081-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:29.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:29.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:29 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:30.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[29291]: pgmap v254: 323 pgs: 1 peering, 32 unknown, 290 active+clean; 586 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:18:30.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:30.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:30.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:30.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[29291]: osdmap e198: 8 total, 8 up, 8 in 2024-03-12T03:18:30.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/521707840' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:30.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:30.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[40484]: pgmap v254: 323 pgs: 1 peering, 32 unknown, 290 active+clean; 586 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:18:30.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:30.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:30.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:30.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[40484]: osdmap e198: 8 total, 8 up, 8 in 2024-03-12T03:18:30.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/521707840' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:30.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:30.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:30 smithi151 ceph-mon[32923]: pgmap v254: 323 pgs: 1 peering, 32 unknown, 290 active+clean; 586 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:18:30.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:30.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:30 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:30.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:30.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:30 smithi151 ceph-mon[32923]: osdmap e198: 8 total, 8 up, 8 in 2024-03-12T03:18:30.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/521707840' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:30.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:30 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:30.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:18:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:18:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:18:31.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:31.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[29291]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi094-78081-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi094-78081-32"}]': finished 2024-03-12T03:18:31.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/521707840' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:31.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:31.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[29291]: osdmap e199: 8 total, 8 up, 8 in 2024-03-12T03:18:31.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-21"}]: dispatch 2024-03-12T03:18:31.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-21"}]: dispatch 2024-03-12T03:18:31.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:31.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[40484]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi094-78081-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi094-78081-32"}]': finished 2024-03-12T03:18:31.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/521707840' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:31.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:31.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[40484]: osdmap e199: 8 total, 8 up, 8 in 2024-03-12T03:18:31.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-21"}]: dispatch 2024-03-12T03:18:31.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:31 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-21"}]: dispatch 2024-03-12T03:18:31.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:31.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:31 smithi151 ceph-mon[32923]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi094-78081-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi094-78081-32"}]': finished 2024-03-12T03:18:31.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/521707840' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:31.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:31 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:31.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:31 smithi151 ceph-mon[32923]: osdmap e199: 8 total, 8 up, 8 in 2024-03-12T03:18:31.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-21"}]: dispatch 2024-03-12T03:18:31.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:31 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-21"}]: dispatch 2024-03-12T03:18:32.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[29291]: pgmap v257: 363 pgs: 1 peering, 72 unknown, 290 active+clean; 586 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:18:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-21"}]': finished 2024-03-12T03:18:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[29291]: osdmap e200: 8 total, 8 up, 8 in 2024-03-12T03:18:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-21", "mode": "writeback"}]: dispatch 2024-03-12T03:18:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-21", "mode": "writeback"}]: dispatch 2024-03-12T03:18:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2104104043' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[29291]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:32.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[40484]: pgmap v257: 363 pgs: 1 peering, 72 unknown, 290 active+clean; 586 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:18:32.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:32.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-21"}]': finished 2024-03-12T03:18:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[40484]: osdmap e200: 8 total, 8 up, 8 in 2024-03-12T03:18:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-21", "mode": "writeback"}]: dispatch 2024-03-12T03:18:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-21", "mode": "writeback"}]: dispatch 2024-03-12T03:18:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2104104043' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:32 smithi094 ceph-mon[40484]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:32.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:32 smithi151 ceph-mon[32923]: pgmap v257: 363 pgs: 1 peering, 72 unknown, 290 active+clean; 586 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:18:32.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:32.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:32 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-21"}]': finished 2024-03-12T03:18:32.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:32 smithi151 ceph-mon[32923]: osdmap e200: 8 total, 8 up, 8 in 2024-03-12T03:18:32.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-21", "mode": "writeback"}]: dispatch 2024-03-12T03:18:32.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:32 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-21", "mode": "writeback"}]: dispatch 2024-03-12T03:18:32.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2104104043' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:32.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:32 smithi151 ceph-mon[32923]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:33.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:33.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:33.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-21", "mode": "writeback"}]': finished 2024-03-12T03:18:33.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[29291]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:33.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[29291]: osdmap e201: 8 total, 8 up, 8 in 2024-03-12T03:18:33.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:33.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[29291]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:33.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:33.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:33.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:33.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:33.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-21", "mode": "writeback"}]': finished 2024-03-12T03:18:33.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[40484]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:33.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[40484]: osdmap e201: 8 total, 8 up, 8 in 2024-03-12T03:18:33.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:33.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[40484]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:33.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:33.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:33 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:33.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:33.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:33 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:33.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:33 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-21", "mode": "writeback"}]': finished 2024-03-12T03:18:33.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:33 smithi151 ceph-mon[32923]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:33.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:33 smithi151 ceph-mon[32923]: osdmap e201: 8 total, 8 up, 8 in 2024-03-12T03:18:33.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:33.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:33 smithi151 ceph-mon[32923]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:33.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:33.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:33 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: pgmap v260: 387 pgs: 1 peering, 87 unknown, 299 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:18:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]': finished 2024-03-12T03:18:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21"}]: dispatch 2024-03-12T03:18:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: osdmap e202: 8 total, 8 up, 8 in 2024-03-12T03:18:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3857684184' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: from='client.56810 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21"}]: dispatch 2024-03-12T03:18:34.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:18:34.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: pgmap v260: 387 pgs: 1 peering, 87 unknown, 299 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:18:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]': finished 2024-03-12T03:18:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21"}]: dispatch 2024-03-12T03:18:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: osdmap e202: 8 total, 8 up, 8 in 2024-03-12T03:18:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:34.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:34.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3857684184' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:34.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: from='client.56810 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:34.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21"}]: dispatch 2024-03-12T03:18:34.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:18:34.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: pgmap v260: 387 pgs: 1 peering, 87 unknown, 299 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi094-78081-32"}]': finished 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21"}]: dispatch 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: osdmap e202: 8 total, 8 up, 8 in 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1827598706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]: dispatch 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3857684184' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: from='client.56810 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21"}]: dispatch 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:18:34.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[29291]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]': finished 2024-03-12T03:18:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[29291]: from='client.56810 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21"}]': finished 2024-03-12T03:18:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[29291]: osdmap e203: 8 total, 8 up, 8 in 2024-03-12T03:18:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[29291]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[29291]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:35.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi094-78081-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:35.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[29291]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi094-78081-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:35.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:35.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:35.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[40484]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]': finished 2024-03-12T03:18:35.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[40484]: from='client.56810 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:35.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21"}]': finished 2024-03-12T03:18:35.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[40484]: osdmap e203: 8 total, 8 up, 8 in 2024-03-12T03:18:35.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:35.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[40484]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:35.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:35.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[40484]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:35.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi094-78081-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:35.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:35 smithi094 ceph-mon[40484]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi094-78081-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:35.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:35.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:35 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:35.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:35 smithi151 ceph-mon[32923]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi094-78081-32"}]': finished 2024-03-12T03:18:35.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:35 smithi151 ceph-mon[32923]: from='client.56810 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi094-78099-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:35.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:35 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-21"}]': finished 2024-03-12T03:18:35.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:35 smithi151 ceph-mon[32923]: osdmap e203: 8 total, 8 up, 8 in 2024-03-12T03:18:35.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:35.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:35 smithi151 ceph-mon[32923]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:35.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:35.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:35 smithi151 ceph-mon[32923]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:35.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi094-78081-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:35.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:35 smithi151 ceph-mon[32923]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi094-78081-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:36 smithi094 ceph-mon[40484]: pgmap v263: 419 pgs: 32 unknown, 387 active+clean; 586 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-12T03:18:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:36 smithi094 ceph-mon[40484]: from='client.49009 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi094-78081-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:36 smithi094 ceph-mon[40484]: osdmap e204: 8 total, 8 up, 8 in 2024-03-12T03:18:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi094-78081-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:36 smithi094 ceph-mon[40484]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi094-78081-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:36 smithi094 ceph-mon[29291]: pgmap v263: 419 pgs: 32 unknown, 387 active+clean; 586 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-12T03:18:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:36.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:36 smithi094 ceph-mon[29291]: from='client.49009 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi094-78081-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:36.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:36 smithi094 ceph-mon[29291]: osdmap e204: 8 total, 8 up, 8 in 2024-03-12T03:18:36.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi094-78081-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:36.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:36 smithi094 ceph-mon[29291]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi094-78081-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:36 smithi151 ceph-mon[32923]: pgmap v263: 419 pgs: 32 unknown, 387 active+clean; 586 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-12T03:18:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:36 smithi151 ceph-mon[32923]: from='client.49009 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi094-78081-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:36 smithi151 ceph-mon[32923]: osdmap e204: 8 total, 8 up, 8 in 2024-03-12T03:18:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi094-78081-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:36.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:36 smithi151 ceph-mon[32923]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi094-78081-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:37.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:37.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:37 smithi094 ceph-mon[29291]: osdmap e205: 8 total, 8 up, 8 in 2024-03-12T03:18:37.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:37.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:37 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:37.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:37.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:37 smithi094 ceph-mon[40484]: osdmap e205: 8 total, 8 up, 8 in 2024-03-12T03:18:37.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:37.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:37 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:37.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:37.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:37 smithi151 ceph-mon[32923]: osdmap e205: 8 total, 8 up, 8 in 2024-03-12T03:18:37.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:37.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:37 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:38.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:18:37 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:18:38.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:38 smithi094 ceph-mon[29291]: pgmap v266: 355 pgs: 32 unknown, 323 active+clean; 586 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-12T03:18:38.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:38.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:38 smithi094 ceph-mon[29291]: from='client.49009 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi094-78081-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi094-78081-33"}]': finished 2024-03-12T03:18:38.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:38 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:38.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:38 smithi094 ceph-mon[29291]: osdmap e206: 8 total, 8 up, 8 in 2024-03-12T03:18:38.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:38 smithi094 ceph-mon[40484]: pgmap v266: 355 pgs: 32 unknown, 323 active+clean; 586 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-12T03:18:38.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:38.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:38 smithi094 ceph-mon[40484]: from='client.49009 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi094-78081-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi094-78081-33"}]': finished 2024-03-12T03:18:38.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:38 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:38.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:38 smithi094 ceph-mon[40484]: osdmap e206: 8 total, 8 up, 8 in 2024-03-12T03:18:38.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:38 smithi151 ceph-mon[32923]: pgmap v266: 355 pgs: 32 unknown, 323 active+clean; 586 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-12T03:18:38.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:38.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:38 smithi151 ceph-mon[32923]: from='client.49009 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi094-78081-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi094-78081-33"}]': finished 2024-03-12T03:18:38.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:38 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:38.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:38 smithi151 ceph-mon[32923]: osdmap e206: 8 total, 8 up, 8 in 2024-03-12T03:18:39.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:39.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:39.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[40484]: osdmap e207: 8 total, 8 up, 8 in 2024-03-12T03:18:39.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1621340236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:39.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[40484]: from='client.49012 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:39.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:39.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[29291]: osdmap e207: 8 total, 8 up, 8 in 2024-03-12T03:18:39.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1621340236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:39.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[29291]: from='client.49012 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:39.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:39.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:39 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:39.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:39.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:39.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:39 smithi151 ceph-mon[32923]: osdmap e207: 8 total, 8 up, 8 in 2024-03-12T03:18:39.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1621340236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:39.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:39 smithi151 ceph-mon[32923]: from='client.49012 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:39.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:39.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:39 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:40.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[29291]: pgmap v269: 363 pgs: 32 unknown, 8 creating+peering, 323 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:18:40.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:40.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[29291]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:40.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[29291]: from='client.49012 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:40.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:40.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[29291]: osdmap e208: 8 total, 8 up, 8 in 2024-03-12T03:18:40.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-23"}]: dispatch 2024-03-12T03:18:40.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-23"}]: dispatch 2024-03-12T03:18:40.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:40.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[29291]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:40.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[40484]: pgmap v269: 363 pgs: 32 unknown, 8 creating+peering, 323 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:18:40.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:40.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[40484]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:40.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[40484]: from='client.49012 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:40.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:40.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[40484]: osdmap e208: 8 total, 8 up, 8 in 2024-03-12T03:18:40.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-23"}]: dispatch 2024-03-12T03:18:40.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-23"}]: dispatch 2024-03-12T03:18:40.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:40.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-mon[40484]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:40.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:40 smithi151 ceph-mon[32923]: pgmap v269: 363 pgs: 32 unknown, 8 creating+peering, 323 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:18:40.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:40.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:40 smithi151 ceph-mon[32923]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:40.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:40 smithi151 ceph-mon[32923]: from='client.49012 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:40.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:40 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:40.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:40 smithi151 ceph-mon[32923]: osdmap e208: 8 total, 8 up, 8 in 2024-03-12T03:18:40.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-23"}]: dispatch 2024-03-12T03:18:40.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:40 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-23"}]: dispatch 2024-03-12T03:18:40.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:40.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:40 smithi151 ceph-mon[32923]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:40.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:18:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:18:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:18:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-23"}]': finished 2024-03-12T03:18:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[29291]: from='client.49009 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]': finished 2024-03-12T03:18:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[29291]: osdmap e209: 8 total, 8 up, 8 in 2024-03-12T03:18:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-23", "mode": "writeback"}]: dispatch 2024-03-12T03:18:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[29291]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3534676730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-23", "mode": "writeback"}]: dispatch 2024-03-12T03:18:41.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[29291]: from='client.56828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-23"}]': finished 2024-03-12T03:18:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[40484]: from='client.49009 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]': finished 2024-03-12T03:18:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[40484]: osdmap e209: 8 total, 8 up, 8 in 2024-03-12T03:18:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-23", "mode": "writeback"}]: dispatch 2024-03-12T03:18:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[40484]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3534676730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-23", "mode": "writeback"}]: dispatch 2024-03-12T03:18:41.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:41 smithi094 ceph-mon[40484]: from='client.56828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:41.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:41.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:41 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-23"}]': finished 2024-03-12T03:18:41.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:41 smithi151 ceph-mon[32923]: from='client.49009 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi094-78081-33"}]': finished 2024-03-12T03:18:41.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:41 smithi151 ceph-mon[32923]: osdmap e209: 8 total, 8 up, 8 in 2024-03-12T03:18:41.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1618260776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:41.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-23", "mode": "writeback"}]: dispatch 2024-03-12T03:18:41.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:41 smithi151 ceph-mon[32923]: from='client.49009 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]: dispatch 2024-03-12T03:18:41.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3534676730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:41.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:41 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-23", "mode": "writeback"}]: dispatch 2024-03-12T03:18:41.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:41 smithi151 ceph-mon[32923]: from='client.56828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[29291]: pgmap v272: 387 pgs: 64 unknown, 323 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-12T03:18:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[29291]: from='client.49009 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]': finished 2024-03-12T03:18:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-23", "mode": "writeback"}]': finished 2024-03-12T03:18:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[29291]: from='client.56828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[29291]: osdmap e210: 8 total, 8 up, 8 in 2024-03-12T03:18:42.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:42.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[29291]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:42.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:42.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[29291]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:42.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi094-78081-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:42.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[29291]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi094-78081-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:42.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[40484]: pgmap v272: 387 pgs: 64 unknown, 323 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-12T03:18:42.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:42.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:42.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[40484]: from='client.49009 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]': finished 2024-03-12T03:18:42.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-23", "mode": "writeback"}]': finished 2024-03-12T03:18:42.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[40484]: from='client.56828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:42.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[40484]: osdmap e210: 8 total, 8 up, 8 in 2024-03-12T03:18:42.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:42.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[40484]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:42.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:42.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[40484]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:42.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi094-78081-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:42.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:42 smithi094 ceph-mon[40484]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi094-78081-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:42.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:42 smithi151 ceph-mon[32923]: pgmap v272: 387 pgs: 64 unknown, 323 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-12T03:18:42.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:42.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:42 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:42.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:42 smithi151 ceph-mon[32923]: from='client.49009 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi094-78081-33"}]': finished 2024-03-12T03:18:42.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:42 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-23", "mode": "writeback"}]': finished 2024-03-12T03:18:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:42 smithi151 ceph-mon[32923]: from='client.56828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:42 smithi151 ceph-mon[32923]: osdmap e210: 8 total, 8 up, 8 in 2024-03-12T03:18:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:42 smithi151 ceph-mon[32923]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:42 smithi151 ceph-mon[32923]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi094-78081-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:42 smithi151 ceph-mon[32923]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi094-78081-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:43.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:43.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[40484]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi094-78081-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:43.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[40484]: osdmap e211: 8 total, 8 up, 8 in 2024-03-12T03:18:43.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/164835565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:43.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi094-78081-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:43.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[40484]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi094-78081-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:43.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:43.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:43.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:43.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[29291]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi094-78081-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:43.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[29291]: osdmap e211: 8 total, 8 up, 8 in 2024-03-12T03:18:43.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/164835565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:43.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi094-78081-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:43.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[29291]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi094-78081-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:43.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:43.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:43.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:43.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:43 smithi151 ceph-mon[32923]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi094-78081-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:43.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:43 smithi151 ceph-mon[32923]: osdmap e211: 8 total, 8 up, 8 in 2024-03-12T03:18:43.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/164835565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:43.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi094-78081-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:43.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:43 smithi151 ceph-mon[32923]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi094-78081-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:43.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:43.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:44.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[40484]: pgmap v275: 419 pgs: 6 creating+peering, 83 unknown, 330 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:18:44.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:44.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/164835565' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:44.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:44.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23"}]: dispatch 2024-03-12T03:18:44.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[40484]: osdmap e212: 8 total, 8 up, 8 in 2024-03-12T03:18:44.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23"}]: dispatch 2024-03-12T03:18:44.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[29291]: pgmap v275: 419 pgs: 6 creating+peering, 83 unknown, 330 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:18:44.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:44.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/164835565' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:44.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:44.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23"}]: dispatch 2024-03-12T03:18:44.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[29291]: osdmap e212: 8 total, 8 up, 8 in 2024-03-12T03:18:44.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:44 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23"}]: dispatch 2024-03-12T03:18:44.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:44 smithi151 ceph-mon[32923]: pgmap v275: 419 pgs: 6 creating+peering, 83 unknown, 330 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:18:44.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:44.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/164835565' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:44.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:44 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:44.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23"}]: dispatch 2024-03-12T03:18:44.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:44 smithi151 ceph-mon[32923]: osdmap e212: 8 total, 8 up, 8 in 2024-03-12T03:18:44.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:44 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23"}]: dispatch 2024-03-12T03:18:45.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[40484]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi094-78081-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi094-78081-34"}]': finished 2024-03-12T03:18:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23"}]': finished 2024-03-12T03:18:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[40484]: osdmap e213: 8 total, 8 up, 8 in 2024-03-12T03:18:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3188346795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:45.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[40484]: from='client.56846 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:45.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:45.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:45.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[29291]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi094-78081-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi094-78081-34"}]': finished 2024-03-12T03:18:45.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23"}]': finished 2024-03-12T03:18:45.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[29291]: osdmap e213: 8 total, 8 up, 8 in 2024-03-12T03:18:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3188346795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:45.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:45 smithi094 ceph-mon[29291]: from='client.56846 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:45 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:45.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:45 smithi151 ceph-mon[32923]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi094-78081-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi094-78081-34"}]': finished 2024-03-12T03:18:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:45 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-23"}]': finished 2024-03-12T03:18:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:45 smithi151 ceph-mon[32923]: osdmap e213: 8 total, 8 up, 8 in 2024-03-12T03:18:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3188346795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:45.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:45 smithi151 ceph-mon[32923]: from='client.56846 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:46.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:46 smithi094 ceph-mon[29291]: pgmap v278: 459 pgs: 40 unknown, 32 creating+peering, 387 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-12T03:18:46.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:46.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:46 smithi094 ceph-mon[29291]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:46.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:46 smithi094 ceph-mon[29291]: from='client.56846 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:46.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:46 smithi094 ceph-mon[29291]: osdmap e214: 8 total, 8 up, 8 in 2024-03-12T03:18:46.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:46 smithi094 ceph-mon[40484]: pgmap v278: 459 pgs: 40 unknown, 32 creating+peering, 387 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-12T03:18:46.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:46.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:46 smithi094 ceph-mon[40484]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:46.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:46 smithi094 ceph-mon[40484]: from='client.56846 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:46.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:46 smithi094 ceph-mon[40484]: osdmap e214: 8 total, 8 up, 8 in 2024-03-12T03:18:46.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:46 smithi151 ceph-mon[32923]: pgmap v278: 459 pgs: 40 unknown, 32 creating+peering, 387 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-12T03:18:46.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:46.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:46 smithi151 ceph-mon[32923]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:46.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:46 smithi151 ceph-mon[32923]: from='client.56846 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:46.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:46 smithi151 ceph-mon[32923]: osdmap e214: 8 total, 8 up, 8 in 2024-03-12T03:18:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[29291]: osdmap e215: 8 total, 8 up, 8 in 2024-03-12T03:18:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[29291]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/724699387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:47.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[29291]: from='client.56852 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:47.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:47.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[40484]: osdmap e215: 8 total, 8 up, 8 in 2024-03-12T03:18:47.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:47.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:47.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:47.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[40484]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:47.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/724699387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:47.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:47 smithi094 ceph-mon[40484]: from='client.56852 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:47 smithi151 ceph-mon[32923]: osdmap e215: 8 total, 8 up, 8 in 2024-03-12T03:18:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:47.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:47.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:47 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:47.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:47 smithi151 ceph-mon[32923]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:47.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/724699387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:47.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:47 smithi151 ceph-mon[32923]: from='client.56852 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:48.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:18:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:18:48.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: pgmap v281: 483 pgs: 96 unknown, 32 creating+peering, 355 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 1023 B/s wr, 1 op/s 2024-03-12T03:18:48.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:48.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:48.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]': finished 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.56852 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: osdmap e216: 8 total, 8 up, 8 in 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]': finished 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: osdmap e217: 8 total, 8 up, 8 in 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi094-78081-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:48.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[29291]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi094-78081-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:48.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: pgmap v281: 483 pgs: 96 unknown, 32 creating+peering, 355 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 1023 B/s wr, 1 op/s 2024-03-12T03:18:48.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:48.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:48.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]': finished 2024-03-12T03:18:48.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.56852 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:48.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: osdmap e216: 8 total, 8 up, 8 in 2024-03-12T03:18:48.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:48.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:48.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]': finished 2024-03-12T03:18:48.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: osdmap e217: 8 total, 8 up, 8 in 2024-03-12T03:18:48.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:48.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:48.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:48.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:48.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:48.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:48.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi094-78081-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:48.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:48 smithi094 ceph-mon[40484]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi094-78081-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:48.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: pgmap v281: 483 pgs: 96 unknown, 32 creating+peering, 355 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 1023 B/s wr, 1 op/s 2024-03-12T03:18:48.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:48.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:48.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi094-78081-34"}]': finished 2024-03-12T03:18:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.56852 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi094-78099-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: osdmap e216: 8 total, 8 up, 8 in 2024-03-12T03:18:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1533723097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]: dispatch 2024-03-12T03:18:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi094-78081-34"}]': finished 2024-03-12T03:18:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: osdmap e217: 8 total, 8 up, 8 in 2024-03-12T03:18:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi094-78081-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:48.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:48 smithi151 ceph-mon[32923]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi094-78081-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:49.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:49.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:49.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-12T03:18:49.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-12T03:18:49.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:18:49.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:49.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:49.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: from='client.56858 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi094-78081-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:49.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-12T03:18:49.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: osdmap e218: 8 total, 8 up, 8 in 2024-03-12T03:18:49.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-25"}]: dispatch 2024-03-12T03:18:49.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi094-78081-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:49.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi094-78081-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:49.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-25"}]: dispatch 2024-03-12T03:18:49.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:49.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:49.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-12T03:18:49.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-12T03:18:49.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:18:49.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:49.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:49.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: from='client.56858 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi094-78081-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:49.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-12T03:18:49.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: osdmap e218: 8 total, 8 up, 8 in 2024-03-12T03:18:49.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-25"}]: dispatch 2024-03-12T03:18:49.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi094-78081-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:49.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi094-78081-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:49.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-25"}]: dispatch 2024-03-12T03:18:49.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:49.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:49.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-12T03:18:49.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-12T03:18:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:18:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: from='client.56858 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi094-78081-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-12T03:18:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: osdmap e218: 8 total, 8 up, 8 in 2024-03-12T03:18:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-25"}]: dispatch 2024-03-12T03:18:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi094-78081-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi094-78081-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:49.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-25"}]: dispatch 2024-03-12T03:18:50.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:18:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:18:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:18:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:50 smithi094 ceph-mon[29291]: pgmap v284: 451 pgs: 451 active+clean; 586 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:50 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-25"}]': finished 2024-03-12T03:18:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:50 smithi094 ceph-mon[40484]: pgmap v284: 451 pgs: 451 active+clean; 586 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:50 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-25"}]': finished 2024-03-12T03:18:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:50 smithi151 ceph-mon[32923]: pgmap v284: 451 pgs: 451 active+clean; 586 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:50 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-25"}]': finished 2024-03-12T03:18:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-25", "mode": "writeback"}]: dispatch 2024-03-12T03:18:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: osdmap e219: 8 total, 8 up, 8 in 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-25", "mode": "writeback"}]: dispatch 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "29"}]: dispatch 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "29"}]: dispatch 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='client.56858 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi094-78081-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi094-78081-35"}]': finished 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-25", "mode": "writeback"}]': finished 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "29"}]': finished 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: osdmap e220: 8 total, 8 up, 8 in 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi094-78177-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi094-78177-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:51.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-25", "mode": "writeback"}]: dispatch 2024-03-12T03:18:51.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:51.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: osdmap e219: 8 total, 8 up, 8 in 2024-03-12T03:18:51.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-25", "mode": "writeback"}]: dispatch 2024-03-12T03:18:51.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "29"}]: dispatch 2024-03-12T03:18:51.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "29"}]: dispatch 2024-03-12T03:18:51.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:51.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='client.56858 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi094-78081-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi094-78081-35"}]': finished 2024-03-12T03:18:51.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-25", "mode": "writeback"}]': finished 2024-03-12T03:18:51.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "29"}]': finished 2024-03-12T03:18:51.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: osdmap e220: 8 total, 8 up, 8 in 2024-03-12T03:18:51.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:51.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:51.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:51.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:51.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi094-78177-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:51.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi094-78177-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:51.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:51.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:51 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-25", "mode": "writeback"}]: dispatch 2024-03-12T03:18:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3055638823' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-12T03:18:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: osdmap e219: 8 total, 8 up, 8 in 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-25", "mode": "writeback"}]: dispatch 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "29"}]: dispatch 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "29"}]: dispatch 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='client.56858 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi094-78081-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi094-78081-35"}]': finished 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-25", "mode": "writeback"}]': finished 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi094-78177-1", "var": "pgp_num_actual", "val": "29"}]': finished 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: osdmap e220: 8 total, 8 up, 8 in 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi094-78177-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi094-78177-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:51.716 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:51 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:18:52.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[29291]: pgmap v287: 387 pgs: 387 active+clean; 586 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:18:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[29291]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi094-78177-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[29291]: osdmap e221: 8 total, 8 up, 8 in 2024-03-12T03:18:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25"}]: dispatch 2024-03-12T03:18:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi094-78177-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25"}]: dispatch 2024-03-12T03:18:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[29291]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi094-78177-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[40484]: pgmap v287: 387 pgs: 387 active+clean; 586 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:18:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[40484]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi094-78177-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[40484]: osdmap e221: 8 total, 8 up, 8 in 2024-03-12T03:18:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25"}]: dispatch 2024-03-12T03:18:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi094-78177-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25"}]: dispatch 2024-03-12T03:18:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:52 smithi094 ceph-mon[40484]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi094-78177-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:52 smithi151 ceph-mon[32923]: pgmap v287: 387 pgs: 387 active+clean; 586 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:18:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:52 smithi151 ceph-mon[32923]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi094-78177-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:18:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:52 smithi151 ceph-mon[32923]: osdmap e221: 8 total, 8 up, 8 in 2024-03-12T03:18:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25"}]: dispatch 2024-03-12T03:18:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi094-78177-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25"}]: dispatch 2024-03-12T03:18:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:52 smithi151 ceph-mon[32923]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi094-78177-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:53.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25"}]': finished 2024-03-12T03:18:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[40484]: osdmap e222: 8 total, 8 up, 8 in 2024-03-12T03:18:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2952024674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi094-78099-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[40484]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[40484]: from='client.56870 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi094-78099-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:53.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25"}]': finished 2024-03-12T03:18:53.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[29291]: osdmap e222: 8 total, 8 up, 8 in 2024-03-12T03:18:53.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:53.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2952024674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi094-78099-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:53.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[29291]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:53.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:53 smithi094 ceph-mon[29291]: from='client.56870 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi094-78099-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:53 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:53 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:18:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:53 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-25"}]': finished 2024-03-12T03:18:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:53 smithi151 ceph-mon[32923]: osdmap e222: 8 total, 8 up, 8 in 2024-03-12T03:18:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2952024674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi094-78099-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:53 smithi151 ceph-mon[32923]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:53 smithi151 ceph-mon[32923]: from='client.56870 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi094-78099-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[29291]: pgmap v290: 300 pgs: 8 unknown, 292 active+clean; 584 KiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[29291]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi094-78177-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi094-78177-2"}]': finished 2024-03-12T03:18:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[29291]: from='client.56858 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]': finished 2024-03-12T03:18:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[29291]: from='client.56870 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi094-78099-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[29291]: osdmap e223: 8 total, 8 up, 8 in 2024-03-12T03:18:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:54.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[29291]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[40484]: pgmap v290: 300 pgs: 8 unknown, 292 active+clean; 584 KiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[40484]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi094-78177-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi094-78177-2"}]': finished 2024-03-12T03:18:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[40484]: from='client.56858 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]': finished 2024-03-12T03:18:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[40484]: from='client.56870 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi094-78099-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[40484]: osdmap e223: 8 total, 8 up, 8 in 2024-03-12T03:18:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:54 smithi094 ceph-mon[40484]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:54 smithi151 ceph-mon[32923]: pgmap v290: 300 pgs: 8 unknown, 292 active+clean; 584 KiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:18:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:54 smithi151 ceph-mon[32923]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi094-78177-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi094-78177-2"}]': finished 2024-03-12T03:18:54.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:54 smithi151 ceph-mon[32923]: from='client.56858 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi094-78081-35"}]': finished 2024-03-12T03:18:54.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:54 smithi151 ceph-mon[32923]: from='client.56870 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi094-78099-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:54.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:54 smithi151 ceph-mon[32923]: osdmap e223: 8 total, 8 up, 8 in 2024-03-12T03:18:54.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/124476038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:54.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:54 smithi151 ceph-mon[32923]: from='client.56858 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]: dispatch 2024-03-12T03:18:55.261 INFO:tasks.workunit.client.0.smithi094.stdout:st: : entry=9 expected=9 2024-03-12T03:18:55.261 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:de5d7c5f:::12:head 2024-03-12T03:18:55.261 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:de5d7c5f:::12:head expected=16:de5d7c5f:::12:head 2024-03-12T03:18:55.261 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:de5d7c5f:::12:head -> 12 2024-03-12T03:18:55.261 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=12 expected=12 2024-03-12T03:18:55.261 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:d83876eb:::4:head 2024-03-12T03:18:55.261 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:d83876eb:::4:head expected=16:d83876eb:::4:head 2024-03-12T03:18:55.261 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:d83876eb:::4:head -> 4 2024-03-12T03:18:55.261 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=4 expected=4 2024-03-12T03:18:55.261 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:cfc208b3:::3:head 2024-03-12T03:18:55.261 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:cfc208b3:::3:head expected=16:cfc208b3:::3:head 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:cfc208b3:::3:head -> 3 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=3 expected=3 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:c4fdafeb:::6:head 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:c4fdafeb:::6:head expected=16:c4fdafeb:::6:head 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:c4fdafeb:::6:head -> 6 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=6 expected=6 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:bd63b0f1:::8:head 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:bd63b0f1:::8:head expected=16:bd63b0f1:::8:head 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:bd63b0f1:::8:head -> 8 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=8 expected=8 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:b29083e3:::5:head 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:b29083e3:::5:head expected=16:b29083e3:::5:head 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:b29083e3:::5:head -> 5 2024-03-12T03:18:55.262 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=5 expected=5 2024-03-12T03:18:55.263 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:89d3ae78:::11:head 2024-03-12T03:18:55.263 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:89d3ae78:::11:head expected=16:89d3ae78:::11:head 2024-03-12T03:18:55.263 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:89d3ae78:::11:head -> 11 2024-03-12T03:18:55.263 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=11 expected=11 2024-03-12T03:18:55.263 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : seek to 16:02547ec2:::1:head 2024-03-12T03:18:55.263 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : cursor()=16:02547ec2:::1:head expected=16:02547ec2:::1:head 2024-03-12T03:18:55.263 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: > 16:02547ec2:::1:head -> 1 2024-03-12T03:18:55.263 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: : entry=1 expected=1 2024-03-12T03:18:55.263 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (227 ms) 2024-03-12T03:18:55.263 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-12T03:18:55.263 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (112965 ms) 2024-03-12T03:18:55.263 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-12T03:18:55.263 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: split 0/5 -> MIN 16:33333333::::head 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: split 1/5 -> 16:33333333::::head 16:66666666::::head 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: split 2/5 -> 16:66666666::::head 16:99999999::::head 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: split 3/5 -> 16:99999999::::head 16:cccccccc::::head 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: split 4/5 -> 16:cccccccc::::head MAX 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (30032 ms) 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [----------] 7 tests from LibRadosList (143761 ms total) 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1043 ms) 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: myset foo1,foo2,foo3 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo1 2024-03-12T03:18:55.264 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo2 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo3 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: myset foo1,foo4,foo5 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo4 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo5 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo1 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: myset foo6,foo7 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo7 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: foo6 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: ns1:foo4 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: ns1:foo5 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: ns2:foo7 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: ns2:foo6 2024-03-12T03:18:55.265 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: ns1:foo1 2024-03-12T03:18:55.266 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: :foo1 2024-03-12T03:18:55.266 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: :foo2 2024-03-12T03:18:55.266 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: :foo3 2024-03-12T03:18:55.266 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (50 ms) 2024-03-12T03:18:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[29291]: pgmap v293: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:18:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[29291]: from='client.56858 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]': finished 2024-03-12T03:18:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[29291]: osdmap e224: 8 total, 8 up, 8 in 2024-03-12T03:18:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[29291]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[29291]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi094-78081-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[29291]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi094-78081-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[40484]: pgmap v293: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:18:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[40484]: from='client.56858 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]': finished 2024-03-12T03:18:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[40484]: osdmap e224: 8 total, 8 up, 8 in 2024-03-12T03:18:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[40484]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[40484]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi094-78081-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:56.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:56 smithi094 ceph-mon[40484]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi094-78081-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:56 smithi151 ceph-mon[32923]: pgmap v293: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:18:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:56 smithi151 ceph-mon[32923]: from='client.56858 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi094-78081-35"}]': finished 2024-03-12T03:18:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:56 smithi151 ceph-mon[32923]: osdmap e224: 8 total, 8 up, 8 in 2024-03-12T03:18:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:56 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:56 smithi151 ceph-mon[32923]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:56 smithi151 ceph-mon[32923]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi094-78081-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:56 smithi151 ceph-mon[32923]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi094-78081-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[29291]: from='client.49069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi094-78081-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[29291]: osdmap e225: 8 total, 8 up, 8 in 2024-03-12T03:18:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi094-78081-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2376774489' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi094-78099-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[29291]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi094-78081-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[29291]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[29291]: from='client.49066 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi094-78099-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[29291]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]': finished 2024-03-12T03:18:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[40484]: from='client.49069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi094-78081-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[40484]: osdmap e225: 8 total, 8 up, 8 in 2024-03-12T03:18:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi094-78081-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2376774489' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi094-78099-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[40484]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi094-78081-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[40484]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[40484]: from='client.49066 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi094-78099-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:57 smithi094 ceph-mon[40484]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]': finished 2024-03-12T03:18:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:57 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:57 smithi151 ceph-mon[32923]: from='client.49069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi094-78081-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:57 smithi151 ceph-mon[32923]: osdmap e225: 8 total, 8 up, 8 in 2024-03-12T03:18:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi094-78081-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2376774489' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi094-78099-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:57 smithi151 ceph-mon[32923]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi094-78081-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:18:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:57 smithi151 ceph-mon[32923]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:57 smithi151 ceph-mon[32923]: from='client.49066 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi094-78099-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:57 smithi151 ceph-mon[32923]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi094-78177-2"}]': finished 2024-03-12T03:18:58.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:18:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:18:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: pgmap v296: 324 pgs: 64 unknown, 260 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:18:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: from='client.49066 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi094-78099-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: osdmap e226: 8 total, 8 up, 8 in 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: from='client.49069 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi094-78081-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi094-78081-36"}]': finished 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]': finished 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: osdmap e227: 8 total, 8 up, 8 in 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-27"}]: dispatch 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-27"}]: dispatch 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[29291]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:58.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: pgmap v296: 324 pgs: 64 unknown, 260 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:18:58.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: from='client.49066 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi094-78099-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:58.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:58.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: osdmap e226: 8 total, 8 up, 8 in 2024-03-12T03:18:58.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:58.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:58.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:58.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:58.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: from='client.49069 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi094-78081-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi094-78081-36"}]': finished 2024-03-12T03:18:58.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]': finished 2024-03-12T03:18:58.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:58.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: osdmap e227: 8 total, 8 up, 8 in 2024-03-12T03:18:58.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-27"}]: dispatch 2024-03-12T03:18:58.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-27"}]: dispatch 2024-03-12T03:18:58.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:58.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:58 smithi094 ceph-mon[40484]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: pgmap v296: 324 pgs: 64 unknown, 260 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:18:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: from='client.49066 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi094-78099-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:18:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/839975819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: osdmap e226: 8 total, 8 up, 8 in 2024-03-12T03:18:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]: dispatch 2024-03-12T03:18:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:18:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:18:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: from='client.49069 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi094-78081-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi094-78081-36"}]': finished 2024-03-12T03:18:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi094-78177-2"}]': finished 2024-03-12T03:18:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:18:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: osdmap e227: 8 total, 8 up, 8 in 2024-03-12T03:18:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-27"}]: dispatch 2024-03-12T03:18:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-27"}]: dispatch 2024-03-12T03:18:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:58 smithi151 ceph-mon[32923]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-27"}]': finished 2024-03-12T03:18:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: from='client.49081 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:59.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: osdmap e228: 8 total, 8 up, 8 in 2024-03-12T03:18:59.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-27", "mode": "writeback"}]: dispatch 2024-03-12T03:18:59.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-27", "mode": "writeback"}]: dispatch 2024-03-12T03:18:59.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1353711010' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78177-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:59.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:59.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78177-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:59.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[29291]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:59.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:59.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:59.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:59.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:59.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:59.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-27"}]': finished 2024-03-12T03:18:59.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: from='client.49081 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:59.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: osdmap e228: 8 total, 8 up, 8 in 2024-03-12T03:18:59.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-27", "mode": "writeback"}]: dispatch 2024-03-12T03:18:59.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-27", "mode": "writeback"}]: dispatch 2024-03-12T03:18:59.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1353711010' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78177-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:59.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:59.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78177-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:59.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:18:59 smithi094 ceph-mon[40484]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:18:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:18:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-27"}]': finished 2024-03-12T03:18:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: from='client.49081 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:18:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: osdmap e228: 8 total, 8 up, 8 in 2024-03-12T03:18:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-27", "mode": "writeback"}]: dispatch 2024-03-12T03:18:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-27", "mode": "writeback"}]: dispatch 2024-03-12T03:18:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1353711010' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78177-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:18:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78177-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:18:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:18:59 smithi151 ceph-mon[32923]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:19:00.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:19:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:19:00.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[29291]: pgmap v299: 300 pgs: 8 unknown, 292 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-12T03:19:00.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:00.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-27", "mode": "writeback"}]': finished 2024-03-12T03:19:00.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[29291]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78177-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:00.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1353711010' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi094-78177-3","pool2":"test-rados-api-smithi094-78177-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-12T03:19:00.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[29291]: osdmap e229: 8 total, 8 up, 8 in 2024-03-12T03:19:00.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:19:00.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[29291]: from='client.49078 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi094-78177-3","pool2":"test-rados-api-smithi094-78177-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-12T03:19:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[29291]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:19:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[40484]: pgmap v299: 300 pgs: 8 unknown, 292 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-12T03:19:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-27", "mode": "writeback"}]': finished 2024-03-12T03:19:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[40484]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78177-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1353711010' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi094-78177-3","pool2":"test-rados-api-smithi094-78177-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-12T03:19:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[40484]: osdmap e229: 8 total, 8 up, 8 in 2024-03-12T03:19:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:19:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[40484]: from='client.49078 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi094-78177-3","pool2":"test-rados-api-smithi094-78177-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-12T03:19:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:00 smithi094 ceph-mon[40484]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:19:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:00 smithi151 ceph-mon[32923]: pgmap v299: 300 pgs: 8 unknown, 292 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-12T03:19:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:00 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:00 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-27", "mode": "writeback"}]': finished 2024-03-12T03:19:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:00 smithi151 ceph-mon[32923]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78177-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1353711010' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi094-78177-3","pool2":"test-rados-api-smithi094-78177-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-12T03:19:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:00 smithi151 ceph-mon[32923]: osdmap e229: 8 total, 8 up, 8 in 2024-03-12T03:19:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:19:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:00 smithi151 ceph-mon[32923]: from='client.49078 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi094-78177-3","pool2":"test-rados-api-smithi094-78177-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-12T03:19:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:00 smithi151 ceph-mon[32923]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:19:01.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-03-12T03:19:01.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 1 0 2024-03-12T03:19:01.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 10 0 2024-03-12T03:19:01.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 13 0 2024-03-12T03:19:01.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 7 0 2024-03-12T03:19:01.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 14 0 2024-03-12T03:19:01.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 0 0 2024-03-12T03:19:01.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 15 0 2024-03-12T03:19:01.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 11 0 2024-03-12T03:19:01.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 5 0 2024-03-12T03:19:01.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 8 0 2024-03-12T03:19:01.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 6 0 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 3 0 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 4 0 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 12 0 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 9 0 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 2 0 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (41 ms) 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [----------] 3 tests from LibRadosListEC (1134 ms total) 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3003 ms) 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [----------] 1 test from LibRadosListNP (3003 ms total) 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: 2024-03-12T03:19:01.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [----------] Global test environment tear-down 2024-03-12T03:19:01.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [==========] 11 tests from 3 test suites ran. (164461 ms total) 2024-03-12T03:19:01.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_list: [ PASSED ] 11 tests. 2024-03-12T03:19:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[40484]: pgmap v302: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-12T03:19:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[40484]: from='client.49081 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-46"}]': finished 2024-03-12T03:19:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[40484]: from='client.49078 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi094-78177-3","pool2":"test-rados-api-smithi094-78177-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-12T03:19:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[40484]: from='client.49069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]': finished 2024-03-12T03:19:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[40484]: osdmap e230: 8 total, 8 up, 8 in 2024-03-12T03:19:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:19:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[40484]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:19:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:02.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[29291]: pgmap v302: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-12T03:19:02.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[29291]: from='client.49081 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-46"}]': finished 2024-03-12T03:19:02.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[29291]: from='client.49078 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi094-78177-3","pool2":"test-rados-api-smithi094-78177-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-12T03:19:02.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[29291]: from='client.49069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]': finished 2024-03-12T03:19:02.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[29291]: osdmap e230: 8 total, 8 up, 8 in 2024-03-12T03:19:02.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:19:02.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[29291]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:19:02.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:02.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:02.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:02 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:02.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:02 smithi151 ceph-mon[32923]: pgmap v302: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-12T03:19:02.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:02 smithi151 ceph-mon[32923]: from='client.49081 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-46"}]': finished 2024-03-12T03:19:02.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:02 smithi151 ceph-mon[32923]: from='client.49078 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi094-78177-3","pool2":"test-rados-api-smithi094-78177-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-12T03:19:02.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:02 smithi151 ceph-mon[32923]: from='client.49069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi094-78081-36"}]': finished 2024-03-12T03:19:02.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:02 smithi151 ceph-mon[32923]: osdmap e230: 8 total, 8 up, 8 in 2024-03-12T03:19:02.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2621158008' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:19:02.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:02 smithi151 ceph-mon[32923]: from='client.49069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]: dispatch 2024-03-12T03:19:02.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:02.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:02 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:02.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:02 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: from='client.49069 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]': finished 2024-03-12T03:19:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: osdmap e231: 8 total, 8 up, 8 in 2024-03-12T03:19:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27"}]: dispatch 2024-03-12T03:19:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27"}]: dispatch 2024-03-12T03:19:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:19:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:19:03.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:03.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:03.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:19:03.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:03.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: from='client.49069 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]': finished 2024-03-12T03:19:03.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:03.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: osdmap e231: 8 total, 8 up, 8 in 2024-03-12T03:19:03.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27"}]: dispatch 2024-03-12T03:19:03.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27"}]: dispatch 2024-03-12T03:19:03.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:03.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:03.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:03.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:19:03.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:19:03.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:03.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:03.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:19:03.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:03 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:03.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: from='client.49069 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi094-78081-36"}]': finished 2024-03-12T03:19:03.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:03.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: osdmap e231: 8 total, 8 up, 8 in 2024-03-12T03:19:03.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27"}]: dispatch 2024-03-12T03:19:03.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27"}]: dispatch 2024-03-12T03:19:03.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:03.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:03.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:03.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:19:03.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:19:03.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:03.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:03.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:19:03.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:03 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[40484]: pgmap v305: 300 pgs: 7 unknown, 293 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27"}]': finished 2024-03-12T03:19:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[40484]: osdmap e232: 8 total, 8 up, 8 in 2024-03-12T03:19:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi094-78081-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:19:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[40484]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:19:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[40484]: from='client.49081 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]': finished 2024-03-12T03:19:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[29291]: pgmap v305: 300 pgs: 7 unknown, 293 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27"}]': finished 2024-03-12T03:19:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[29291]: osdmap e232: 8 total, 8 up, 8 in 2024-03-12T03:19:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi094-78081-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:19:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[29291]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:19:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:04 smithi094 ceph-mon[29291]: from='client.49081 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]': finished 2024-03-12T03:19:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:04 smithi151 ceph-mon[32923]: pgmap v305: 300 pgs: 7 unknown, 293 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:04 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:04 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-27"}]': finished 2024-03-12T03:19:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:04 smithi151 ceph-mon[32923]: osdmap e232: 8 total, 8 up, 8 in 2024-03-12T03:19:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi094-78081-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:19:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:04 smithi151 ceph-mon[32923]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:19:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:04 smithi151 ceph-mon[32923]: from='client.49081 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-46"}]': finished 2024-03-12T03:19:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:19:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: osdmap e233: 8 total, 8 up, 8 in 2024-03-12T03:19:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:19:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi094-78081-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi094-78081-37"}]': finished 2024-03-12T03:19:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: from='client.49081 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]': finished 2024-03-12T03:19:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: osdmap e234: 8 total, 8 up, 8 in 2024-03-12T03:19:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:05.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:05.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:05.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:05.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[40484]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:19:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: osdmap e233: 8 total, 8 up, 8 in 2024-03-12T03:19:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:19:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi094-78081-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi094-78081-37"}]': finished 2024-03-12T03:19:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: from='client.49081 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]': finished 2024-03-12T03:19:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: osdmap e234: 8 total, 8 up, 8 in 2024-03-12T03:19:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:05.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:05.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:05.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:05.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:05.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:05.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:05 smithi094 ceph-mon[29291]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1692757592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:19:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: osdmap e233: 8 total, 8 up, 8 in 2024-03-12T03:19:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: from='client.49081 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]: dispatch 2024-03-12T03:19:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi094-78081-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi094-78081-37"}]': finished 2024-03-12T03:19:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: from='client.49081 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-46"}]': finished 2024-03-12T03:19:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: osdmap e234: 8 total, 8 up, 8 in 2024-03-12T03:19:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:05 smithi151 ceph-mon[32923]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:06.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:06 smithi094 ceph-mon[40484]: pgmap v308: 260 pgs: 260 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:06.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:06 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:06.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:06 smithi094 ceph-mon[40484]: from='client.56906 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:06.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:06 smithi094 ceph-mon[40484]: osdmap e235: 8 total, 8 up, 8 in 2024-03-12T03:19:06.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:06.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:06 smithi094 ceph-mon[40484]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:06 smithi094 ceph-mon[29291]: pgmap v308: 260 pgs: 260 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:06 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:06 smithi094 ceph-mon[29291]: from='client.56906 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:06 smithi094 ceph-mon[29291]: osdmap e235: 8 total, 8 up, 8 in 2024-03-12T03:19:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:06 smithi094 ceph-mon[29291]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:06 smithi151 ceph-mon[32923]: pgmap v308: 260 pgs: 260 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:06 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:06 smithi151 ceph-mon[32923]: from='client.56906 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi094-78099-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:06 smithi151 ceph-mon[32923]: osdmap e235: 8 total, 8 up, 8 in 2024-03-12T03:19:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:06 smithi151 ceph-mon[32923]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:08.190 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:19:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:19:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[40484]: pgmap v311: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[40484]: osdmap e236: 8 total, 8 up, 8 in 2024-03-12T03:19:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[40484]: from='client.56906 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-47"}]': finished 2024-03-12T03:19:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37"}]': finished 2024-03-12T03:19:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[40484]: osdmap e237: 8 total, 8 up, 8 in 2024-03-12T03:19:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:08.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:08.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:08.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[29291]: pgmap v311: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:08.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[29291]: osdmap e236: 8 total, 8 up, 8 in 2024-03-12T03:19:08.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:08.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:08.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[29291]: from='client.56906 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-47"}]': finished 2024-03-12T03:19:08.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37"}]': finished 2024-03-12T03:19:08.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[29291]: osdmap e237: 8 total, 8 up, 8 in 2024-03-12T03:19:08.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:08.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:08.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:08 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:08.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:08 smithi151 ceph-mon[32923]: pgmap v311: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:08.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:08 smithi151 ceph-mon[32923]: osdmap e236: 8 total, 8 up, 8 in 2024-03-12T03:19:08.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:08.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:08 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:08.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:08 smithi151 ceph-mon[32923]: from='client.56906 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi094-78099-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi094-78099-47"}]': finished 2024-03-12T03:19:08.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi094-78081-37"}]': finished 2024-03-12T03:19:08.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:08 smithi151 ceph-mon[32923]: osdmap e237: 8 total, 8 up, 8 in 2024-03-12T03:19:08.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi094-78081-37"}]: dispatch 2024-03-12T03:19:08.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:08.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:08 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi094-78081-37"}]': finished 2024-03-12T03:19:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[29291]: osdmap e238: 8 total, 8 up, 8 in 2024-03-12T03:19:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-29"}]: dispatch 2024-03-12T03:19:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-29"}]: dispatch 2024-03-12T03:19:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[29291]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:09.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[29291]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:09.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi094-78081-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:09.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[29291]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi094-78081-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi094-78081-37"}]': finished 2024-03-12T03:19:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[40484]: osdmap e238: 8 total, 8 up, 8 in 2024-03-12T03:19:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-29"}]: dispatch 2024-03-12T03:19:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-29"}]: dispatch 2024-03-12T03:19:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[40484]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:09.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[40484]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:09.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi094-78081-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:09.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:09 smithi094 ceph-mon[40484]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi094-78081-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:09.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:09.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/463843339' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi094-78081-37"}]': finished 2024-03-12T03:19:09.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:09 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:09 smithi151 ceph-mon[32923]: osdmap e238: 8 total, 8 up, 8 in 2024-03-12T03:19:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-29"}]: dispatch 2024-03-12T03:19:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:09 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-29"}]: dispatch 2024-03-12T03:19:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:09 smithi151 ceph-mon[32923]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:09 smithi151 ceph-mon[32923]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi094-78081-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:09.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:09 smithi151 ceph-mon[32923]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi094-78081-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[29291]: pgmap v314: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-12T03:19:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-29"}]': finished 2024-03-12T03:19:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[29291]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi094-78081-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[29291]: osdmap e239: 8 total, 8 up, 8 in 2024-03-12T03:19:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi094-78081-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[29291]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi094-78081-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-29", "mode": "writeback"}]: dispatch 2024-03-12T03:19:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-29", "mode": "writeback"}]: dispatch 2024-03-12T03:19:10.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[29291]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[40484]: pgmap v314: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-12T03:19:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-29"}]': finished 2024-03-12T03:19:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[40484]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi094-78081-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[40484]: osdmap e239: 8 total, 8 up, 8 in 2024-03-12T03:19:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi094-78081-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[40484]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi094-78081-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-29", "mode": "writeback"}]: dispatch 2024-03-12T03:19:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-29", "mode": "writeback"}]: dispatch 2024-03-12T03:19:10.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-mon[40484]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:10.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:10 smithi151 ceph-mon[32923]: pgmap v314: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-12T03:19:10.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:10 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-29"}]': finished 2024-03-12T03:19:10.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:10 smithi151 ceph-mon[32923]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi094-78081-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:10.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:10 smithi151 ceph-mon[32923]: osdmap e239: 8 total, 8 up, 8 in 2024-03-12T03:19:10.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi094-78081-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:10.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:10.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:10 smithi151 ceph-mon[32923]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi094-78081-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:10.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-29", "mode": "writeback"}]: dispatch 2024-03-12T03:19:10.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:10 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-29", "mode": "writeback"}]: dispatch 2024-03-12T03:19:10.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:10 smithi151 ceph-mon[32923]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:10.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:19:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:19:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:19:11.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:11 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:11.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:11 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-29", "mode": "writeback"}]': finished 2024-03-12T03:19:11.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:11 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:11.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:11 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-29", "mode": "writeback"}]': finished 2024-03-12T03:19:11.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:11 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:11.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:11 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-29", "mode": "writeback"}]': finished 2024-03-12T03:19:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[29291]: pgmap v317: 292 pgs: 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-12T03:19:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[29291]: from='client.56906 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]': finished 2024-03-12T03:19:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[29291]: osdmap e240: 8 total, 8 up, 8 in 2024-03-12T03:19:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[29291]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[29291]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi094-78081-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi094-78081-38"}]': finished 2024-03-12T03:19:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[29291]: from='client.56906 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]': finished 2024-03-12T03:19:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[29291]: osdmap e241: 8 total, 8 up, 8 in 2024-03-12T03:19:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:12.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi094-78099-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:12.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[40484]: pgmap v317: 292 pgs: 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-12T03:19:12.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[40484]: from='client.56906 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]': finished 2024-03-12T03:19:12.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:12.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[40484]: osdmap e240: 8 total, 8 up, 8 in 2024-03-12T03:19:12.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[40484]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:12.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[40484]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi094-78081-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi094-78081-38"}]': finished 2024-03-12T03:19:12.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[40484]: from='client.56906 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]': finished 2024-03-12T03:19:12.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[40484]: osdmap e241: 8 total, 8 up, 8 in 2024-03-12T03:19:12.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:12.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:12.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi094-78099-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:12 smithi151 ceph-mon[32923]: pgmap v317: 292 pgs: 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-12T03:19:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:12 smithi151 ceph-mon[32923]: from='client.56906 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi094-78099-47"}]': finished 2024-03-12T03:19:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1259427207' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:12.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:12 smithi151 ceph-mon[32923]: osdmap e240: 8 total, 8 up, 8 in 2024-03-12T03:19:12.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:12 smithi151 ceph-mon[32923]: from='client.56906 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]: dispatch 2024-03-12T03:19:12.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:12 smithi151 ceph-mon[32923]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi094-78081-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi094-78081-38"}]': finished 2024-03-12T03:19:12.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:12 smithi151 ceph-mon[32923]: from='client.56906 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi094-78099-47"}]': finished 2024-03-12T03:19:12.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:12 smithi151 ceph-mon[32923]: osdmap e241: 8 total, 8 up, 8 in 2024-03-12T03:19:12.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:12.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:12.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi094-78099-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:13.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:13 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi094-78099-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:13 smithi094 ceph-mon[29291]: osdmap e242: 8 total, 8 up, 8 in 2024-03-12T03:19:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi094-78099-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:13.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:13 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:13.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi094-78099-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:13.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:13 smithi094 ceph-mon[40484]: osdmap e242: 8 total, 8 up, 8 in 2024-03-12T03:19:13.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi094-78099-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:13.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:13 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:13.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi094-78099-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:13.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:13 smithi151 ceph-mon[32923]: osdmap e242: 8 total, 8 up, 8 in 2024-03-12T03:19:13.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi094-78099-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:14.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:14 smithi094 ceph-mon[29291]: pgmap v320: 300 pgs: 1 creating+peering, 7 unknown, 292 active+clean; 4.6 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:14 smithi094 ceph-mon[29291]: osdmap e243: 8 total, 8 up, 8 in 2024-03-12T03:19:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:14 smithi094 ceph-mon[29291]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:14 smithi094 ceph-mon[40484]: pgmap v320: 300 pgs: 1 creating+peering, 7 unknown, 292 active+clean; 4.6 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:14 smithi094 ceph-mon[40484]: osdmap e243: 8 total, 8 up, 8 in 2024-03-12T03:19:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:14 smithi094 ceph-mon[40484]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:14 smithi151 ceph-mon[32923]: pgmap v320: 300 pgs: 1 creating+peering, 7 unknown, 292 active+clean; 4.6 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:14 smithi151 ceph-mon[32923]: osdmap e243: 8 total, 8 up, 8 in 2024-03-12T03:19:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:14.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:14 smithi151 ceph-mon[32923]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi094-78099-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi094-78099-48"}]': finished 2024-03-12T03:19:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[40484]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]': finished 2024-03-12T03:19:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[40484]: osdmap e244: 8 total, 8 up, 8 in 2024-03-12T03:19:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[40484]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29"}]: dispatch 2024-03-12T03:19:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29"}]: dispatch 2024-03-12T03:19:15.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:15.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:15.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi094-78099-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi094-78099-48"}]': finished 2024-03-12T03:19:15.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[29291]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]': finished 2024-03-12T03:19:15.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:15.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[29291]: osdmap e244: 8 total, 8 up, 8 in 2024-03-12T03:19:15.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:15.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[29291]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:15.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29"}]: dispatch 2024-03-12T03:19:15.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:15 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29"}]: dispatch 2024-03-12T03:19:15.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:15 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:15.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:15 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:15.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:15 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi094-78099-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi094-78099-48"}]': finished 2024-03-12T03:19:15.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:15 smithi151 ceph-mon[32923]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi094-78081-38"}]': finished 2024-03-12T03:19:15.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:15 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:15.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:15 smithi151 ceph-mon[32923]: osdmap e244: 8 total, 8 up, 8 in 2024-03-12T03:19:15.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:15 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/817196916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:15.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:15 smithi151 ceph-mon[32923]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]: dispatch 2024-03-12T03:19:15.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:15 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29"}]: dispatch 2024-03-12T03:19:15.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:15 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29"}]: dispatch 2024-03-12T03:19:16.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[40484]: pgmap v323: 292 pgs: 292 active+clean; 8.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-12T03:19:16.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:16.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[40484]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]': finished 2024-03-12T03:19:16.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29"}]': finished 2024-03-12T03:19:16.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[40484]: osdmap e245: 8 total, 8 up, 8 in 2024-03-12T03:19:16.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:16.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:16.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi094-78081-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:16.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[29291]: pgmap v323: 292 pgs: 292 active+clean; 8.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-12T03:19:16.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:16.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[29291]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]': finished 2024-03-12T03:19:16.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29"}]': finished 2024-03-12T03:19:16.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[29291]: osdmap e245: 8 total, 8 up, 8 in 2024-03-12T03:19:16.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:16.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:16.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi094-78081-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:16.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:16 smithi151 ceph-mon[32923]: pgmap v323: 292 pgs: 292 active+clean; 8.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-12T03:19:16.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:16 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:16.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:16 smithi151 ceph-mon[32923]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi094-78081-38"}]': finished 2024-03-12T03:19:16.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:16 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-29"}]': finished 2024-03-12T03:19:16.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:16 smithi151 ceph-mon[32923]: osdmap e245: 8 total, 8 up, 8 in 2024-03-12T03:19:16.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:16.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:16.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi094-78081-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:18.168 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:19:17 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:19:18.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:18 smithi094 ceph-mon[40484]: pgmap v326: 300 pgs: 8 unknown, 292 active+clean; 8.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-12T03:19:18.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi094-78081-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:18.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:18 smithi094 ceph-mon[40484]: osdmap e246: 8 total, 8 up, 8 in 2024-03-12T03:19:18.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi094-78081-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:18.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:18.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:18 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:18 smithi094 ceph-mon[29291]: pgmap v326: 300 pgs: 8 unknown, 292 active+clean; 8.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-12T03:19:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi094-78081-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:18 smithi094 ceph-mon[29291]: osdmap e246: 8 total, 8 up, 8 in 2024-03-12T03:19:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi094-78081-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:18.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:18 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:18.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:18 smithi151 ceph-mon[32923]: pgmap v326: 300 pgs: 8 unknown, 292 active+clean; 8.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-12T03:19:18.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi094-78081-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:18.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:18 smithi151 ceph-mon[32923]: osdmap e246: 8 total, 8 up, 8 in 2024-03-12T03:19:18.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi094-78081-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:18 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi094-78099-48"}]': finished 2024-03-12T03:19:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[29291]: osdmap e247: 8 total, 8 up, 8 in 2024-03-12T03:19:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:19.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:19.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:19.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi094-78099-48"}]': finished 2024-03-12T03:19:19.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[40484]: osdmap e247: 8 total, 8 up, 8 in 2024-03-12T03:19:19.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:19.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:19.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:19.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:19.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:19.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi094-78099-48"}]': finished 2024-03-12T03:19:19.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:19 smithi151 ceph-mon[32923]: osdmap e247: 8 total, 8 up, 8 in 2024-03-12T03:19:19.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi094-78099-48"}]: dispatch 2024-03-12T03:19:19.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:19.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:19 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:19.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:20.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[29291]: pgmap v329: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:20.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi094-78081-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi094-78081-39"}]': finished 2024-03-12T03:19:20.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi094-78099-48"}]': finished 2024-03-12T03:19:20.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:20.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[29291]: osdmap e248: 8 total, 8 up, 8 in 2024-03-12T03:19:20.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:20.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:20.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:20.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[29291]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:20.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:20.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[29291]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:20.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi094-78099-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:20.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[29291]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi094-78099-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:20.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[40484]: pgmap v329: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:20.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi094-78081-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi094-78081-39"}]': finished 2024-03-12T03:19:20.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi094-78099-48"}]': finished 2024-03-12T03:19:20.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:20.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[40484]: osdmap e248: 8 total, 8 up, 8 in 2024-03-12T03:19:20.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:20.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:20.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:20.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[40484]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:20.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:20.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[40484]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:20.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi094-78099-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:20.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-mon[40484]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi094-78099-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:20.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:20 smithi151 ceph-mon[32923]: pgmap v329: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:20.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi094-78081-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi094-78081-39"}]': finished 2024-03-12T03:19:20.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1102833229' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi094-78099-48"}]': finished 2024-03-12T03:19:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:20 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:20 smithi151 ceph-mon[32923]: osdmap e248: 8 total, 8 up, 8 in 2024-03-12T03:19:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:20 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:20 smithi151 ceph-mon[32923]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:20 smithi151 ceph-mon[32923]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi094-78099-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:20 smithi151 ceph-mon[32923]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi094-78099-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:20.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:19:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:19:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:19:21.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:21.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[40484]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi094-78099-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:21.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-31"}]: dispatch 2024-03-12T03:19:21.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[40484]: osdmap e249: 8 total, 8 up, 8 in 2024-03-12T03:19:21.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-31"}]: dispatch 2024-03-12T03:19:21.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi094-78099-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:21.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[40484]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi094-78099-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:21.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:21.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[29291]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi094-78099-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:21.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-31"}]: dispatch 2024-03-12T03:19:21.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[29291]: osdmap e249: 8 total, 8 up, 8 in 2024-03-12T03:19:21.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-31"}]: dispatch 2024-03-12T03:19:21.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi094-78099-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:21.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:21 smithi094 ceph-mon[29291]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi094-78099-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:21.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:21 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:21.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:21 smithi151 ceph-mon[32923]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi094-78099-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:21.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-31"}]: dispatch 2024-03-12T03:19:21.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:21 smithi151 ceph-mon[32923]: osdmap e249: 8 total, 8 up, 8 in 2024-03-12T03:19:21.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:21 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-31"}]: dispatch 2024-03-12T03:19:21.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi094-78099-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:21.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:21 smithi151 ceph-mon[32923]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi094-78099-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:22.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:22 smithi094 ceph-mon[29291]: pgmap v332: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:22.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:22 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-31"}]': finished 2024-03-12T03:19:22.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-31", "mode": "writeback"}]: dispatch 2024-03-12T03:19:22.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:22 smithi094 ceph-mon[29291]: osdmap e250: 8 total, 8 up, 8 in 2024-03-12T03:19:22.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:22 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-31", "mode": "writeback"}]: dispatch 2024-03-12T03:19:22.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:22.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:22 smithi094 ceph-mon[40484]: pgmap v332: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:22.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:22 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-31"}]': finished 2024-03-12T03:19:22.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-31", "mode": "writeback"}]: dispatch 2024-03-12T03:19:22.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:22 smithi094 ceph-mon[40484]: osdmap e250: 8 total, 8 up, 8 in 2024-03-12T03:19:22.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:22 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-31", "mode": "writeback"}]: dispatch 2024-03-12T03:19:22.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:22.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:22 smithi151 ceph-mon[32923]: pgmap v332: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:22.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:22 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-31"}]': finished 2024-03-12T03:19:22.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-31", "mode": "writeback"}]: dispatch 2024-03-12T03:19:22.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:22 smithi151 ceph-mon[32923]: osdmap e250: 8 total, 8 up, 8 in 2024-03-12T03:19:22.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:22 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-31", "mode": "writeback"}]: dispatch 2024-03-12T03:19:22.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi094-78099-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi094-78099-49"}]': finished 2024-03-12T03:19:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-31", "mode": "writeback"}]': finished 2024-03-12T03:19:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi094-78081-39"}]': finished 2024-03-12T03:19:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: osdmap e251: 8 total, 8 up, 8 in 2024-03-12T03:19:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi094-78081-39"}]': finished 2024-03-12T03:19:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: osdmap e252: 8 total, 8 up, 8 in 2024-03-12T03:19:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31"}]: dispatch 2024-03-12T03:19:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31"}]: dispatch 2024-03-12T03:19:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi094-78081-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:23.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[29291]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi094-78081-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:23.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:23.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi094-78099-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi094-78099-49"}]': finished 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-31", "mode": "writeback"}]': finished 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi094-78081-39"}]': finished 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: osdmap e251: 8 total, 8 up, 8 in 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi094-78081-39"}]': finished 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: osdmap e252: 8 total, 8 up, 8 in 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31"}]: dispatch 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31"}]: dispatch 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:23.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:23.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:23.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi094-78081-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:23.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:23 smithi094 ceph-mon[40484]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi094-78081-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:23.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:23.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi094-78099-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi094-78099-49"}]': finished 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-31", "mode": "writeback"}]': finished 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi094-78081-39"}]': finished 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: osdmap e251: 8 total, 8 up, 8 in 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi094-78081-39"}]: dispatch 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1241730057' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi094-78081-39"}]': finished 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: osdmap e252: 8 total, 8 up, 8 in 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31"}]: dispatch 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31"}]: dispatch 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:23.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:23.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:23.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi094-78081-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:23.466 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:23 smithi151 ceph-mon[32923]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi094-78081-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:24.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[29291]: pgmap v335: 300 pgs: 1 creating+peering, 33 unknown, 266 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:19:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31"}]': finished 2024-03-12T03:19:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[29291]: from='client.56936 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi094-78081-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[29291]: osdmap e253: 8 total, 8 up, 8 in 2024-03-12T03:19:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi094-78081-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[29291]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi094-78081-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[29291]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[40484]: pgmap v335: 300 pgs: 1 creating+peering, 33 unknown, 266 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:19:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31"}]': finished 2024-03-12T03:19:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[40484]: from='client.56936 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi094-78081-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[40484]: osdmap e253: 8 total, 8 up, 8 in 2024-03-12T03:19:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi094-78081-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[40484]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi094-78081-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:24.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:24 smithi094 ceph-mon[40484]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:24 smithi151 ceph-mon[32923]: pgmap v335: 300 pgs: 1 creating+peering, 33 unknown, 266 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:19:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:24 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:24 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-31"}]': finished 2024-03-12T03:19:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:24 smithi151 ceph-mon[32923]: from='client.56936 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi094-78081-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:24 smithi151 ceph-mon[32923]: osdmap e253: 8 total, 8 up, 8 in 2024-03-12T03:19:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:24 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi094-78081-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:24.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:24 smithi151 ceph-mon[32923]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi094-78081-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:24.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:24 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:24.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:24 smithi151 ceph-mon[32923]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:26.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:26 smithi094 ceph-mon[29291]: pgmap v338: 292 pgs: 292 active+clean; 4.6 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:19:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:26 smithi094 ceph-mon[29291]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]': finished 2024-03-12T03:19:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:26 smithi094 ceph-mon[29291]: osdmap e254: 8 total, 8 up, 8 in 2024-03-12T03:19:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:26 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:26 smithi094 ceph-mon[29291]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:26.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:26 smithi094 ceph-mon[40484]: pgmap v338: 292 pgs: 292 active+clean; 4.6 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:19:26.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:26 smithi094 ceph-mon[40484]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]': finished 2024-03-12T03:19:26.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:26.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:26 smithi094 ceph-mon[40484]: osdmap e254: 8 total, 8 up, 8 in 2024-03-12T03:19:26.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:26 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:26.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:26 smithi094 ceph-mon[40484]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:26 smithi151 ceph-mon[32923]: pgmap v338: 292 pgs: 292 active+clean; 4.6 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:19:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:26 smithi151 ceph-mon[32923]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi094-78099-49"}]': finished 2024-03-12T03:19:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/147896090' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:26 smithi151 ceph-mon[32923]: osdmap e254: 8 total, 8 up, 8 in 2024-03-12T03:19:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:26 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:26.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:26 smithi151 ceph-mon[32923]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]: dispatch 2024-03-12T03:19:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[29291]: from='client.56936 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi094-78081-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi094-78081-40"}]': finished 2024-03-12T03:19:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[29291]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]': finished 2024-03-12T03:19:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[29291]: osdmap e255: 8 total, 8 up, 8 in 2024-03-12T03:19:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[29291]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:27.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[29291]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:27.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi094-78099-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:27.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[29291]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi094-78099-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[40484]: from='client.56936 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi094-78081-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi094-78081-40"}]': finished 2024-03-12T03:19:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[40484]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]': finished 2024-03-12T03:19:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[40484]: osdmap e255: 8 total, 8 up, 8 in 2024-03-12T03:19:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[40484]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:27.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:27.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[40484]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:27.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi094-78099-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:27.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:27 smithi094 ceph-mon[40484]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi094-78099-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:27.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:27 smithi151 ceph-mon[32923]: from='client.56936 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi094-78081-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi094-78081-40"}]': finished 2024-03-12T03:19:27.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:27 smithi151 ceph-mon[32923]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi094-78099-49"}]': finished 2024-03-12T03:19:27.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:27 smithi151 ceph-mon[32923]: osdmap e255: 8 total, 8 up, 8 in 2024-03-12T03:19:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:27 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:27 smithi151 ceph-mon[32923]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:27 smithi151 ceph-mon[32923]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi094-78099-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:27.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:27 smithi151 ceph-mon[32923]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi094-78099-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:28.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:19:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:19:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[29291]: pgmap v341: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-12T03:19:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[29291]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi094-78099-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[29291]: osdmap e256: 8 total, 8 up, 8 in 2024-03-12T03:19:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi094-78099-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[29291]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi094-78099-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[40484]: pgmap v341: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-12T03:19:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[40484]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi094-78099-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[40484]: osdmap e256: 8 total, 8 up, 8 in 2024-03-12T03:19:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi094-78099-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:28 smithi094 ceph-mon[40484]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi094-78099-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:28 smithi151 ceph-mon[32923]: pgmap v341: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-12T03:19:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:28 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:28 smithi151 ceph-mon[32923]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi094-78099-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:28 smithi151 ceph-mon[32923]: osdmap e256: 8 total, 8 up, 8 in 2024-03-12T03:19:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi094-78099-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:28 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:28 smithi151 ceph-mon[32923]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi094-78099-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-33"}]: dispatch 2024-03-12T03:19:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: osdmap e257: 8 total, 8 up, 8 in 2024-03-12T03:19:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-33"}]: dispatch 2024-03-12T03:19:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi094-78099-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi094-78099-50"}]': finished 2024-03-12T03:19:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-33"}]': finished 2024-03-12T03:19:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: from='client.56936 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]': finished 2024-03-12T03:19:29.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: osdmap e258: 8 total, 8 up, 8 in 2024-03-12T03:19:29.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-33", "mode": "writeback"}]: dispatch 2024-03-12T03:19:29.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:29.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-33", "mode": "writeback"}]: dispatch 2024-03-12T03:19:29.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[40484]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:29.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:29.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:29.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-33"}]: dispatch 2024-03-12T03:19:29.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: osdmap e257: 8 total, 8 up, 8 in 2024-03-12T03:19:29.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:29.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-33"}]: dispatch 2024-03-12T03:19:29.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:29.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi094-78099-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi094-78099-50"}]': finished 2024-03-12T03:19:29.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-33"}]': finished 2024-03-12T03:19:29.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: from='client.56936 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]': finished 2024-03-12T03:19:29.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: osdmap e258: 8 total, 8 up, 8 in 2024-03-12T03:19:29.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-33", "mode": "writeback"}]: dispatch 2024-03-12T03:19:29.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:29.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-33", "mode": "writeback"}]: dispatch 2024-03-12T03:19:29.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:29 smithi094 ceph-mon[29291]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-33"}]: dispatch 2024-03-12T03:19:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: osdmap e257: 8 total, 8 up, 8 in 2024-03-12T03:19:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-33"}]: dispatch 2024-03-12T03:19:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi094-78099-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi094-78099-50"}]': finished 2024-03-12T03:19:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-33"}]': finished 2024-03-12T03:19:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: from='client.56936 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi094-78081-40"}]': finished 2024-03-12T03:19:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: osdmap e258: 8 total, 8 up, 8 in 2024-03-12T03:19:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-33", "mode": "writeback"}]: dispatch 2024-03-12T03:19:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2293685453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-33", "mode": "writeback"}]: dispatch 2024-03-12T03:19:29.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:29 smithi151 ceph-mon[32923]: from='client.56936 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]: dispatch 2024-03-12T03:19:30.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:19:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:19:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[40484]: pgmap v344: 292 pgs: 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-12T03:19:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-33", "mode": "writeback"}]': finished 2024-03-12T03:19:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[40484]: from='client.56936 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]': finished 2024-03-12T03:19:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[40484]: osdmap e259: 8 total, 8 up, 8 in 2024-03-12T03:19:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi094-78081-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:30.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[29291]: pgmap v344: 292 pgs: 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-12T03:19:30.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:30.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-33", "mode": "writeback"}]': finished 2024-03-12T03:19:30.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[29291]: from='client.56936 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]': finished 2024-03-12T03:19:30.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[29291]: osdmap e259: 8 total, 8 up, 8 in 2024-03-12T03:19:30.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:30.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:30.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi094-78081-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:30.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:30.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:30 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:30 smithi151 ceph-mon[32923]: pgmap v344: 292 pgs: 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-12T03:19:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:30 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:30 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-33", "mode": "writeback"}]': finished 2024-03-12T03:19:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:30 smithi151 ceph-mon[32923]: from='client.56936 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi094-78081-40"}]': finished 2024-03-12T03:19:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:30 smithi151 ceph-mon[32923]: osdmap e259: 8 total, 8 up, 8 in 2024-03-12T03:19:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi094-78081-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:30 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[29291]: pgmap v347: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi094-78081-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[29291]: osdmap e260: 8 total, 8 up, 8 in 2024-03-12T03:19:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi094-78081-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33"}]: dispatch 2024-03-12T03:19:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33"}]: dispatch 2024-03-12T03:19:32.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:32.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[29291]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:32.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:32.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[40484]: pgmap v347: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:32.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi094-78081-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:32.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:32.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[40484]: osdmap e260: 8 total, 8 up, 8 in 2024-03-12T03:19:32.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi094-78081-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:32.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33"}]: dispatch 2024-03-12T03:19:32.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33"}]: dispatch 2024-03-12T03:19:32.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:32.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[40484]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:32.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:32 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:32 smithi151 ceph-mon[32923]: pgmap v347: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi094-78081-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:32 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:32 smithi151 ceph-mon[32923]: osdmap e260: 8 total, 8 up, 8 in 2024-03-12T03:19:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi094-78081-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:32.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33"}]: dispatch 2024-03-12T03:19:32.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:32 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33"}]: dispatch 2024-03-12T03:19:32.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:32.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:32 smithi151 ceph-mon[32923]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:32.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:32 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:33.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33"}]': finished 2024-03-12T03:19:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[40484]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]': finished 2024-03-12T03:19:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[40484]: osdmap e261: 8 total, 8 up, 8 in 2024-03-12T03:19:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[40484]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi094-78081-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi094-78081-41"}]': finished 2024-03-12T03:19:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[40484]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]': finished 2024-03-12T03:19:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[40484]: osdmap e262: 8 total, 8 up, 8 in 2024-03-12T03:19:33.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:33.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33"}]': finished 2024-03-12T03:19:33.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[29291]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]': finished 2024-03-12T03:19:33.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:33.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[29291]: osdmap e261: 8 total, 8 up, 8 in 2024-03-12T03:19:33.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[29291]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:33.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi094-78081-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi094-78081-41"}]': finished 2024-03-12T03:19:33.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[29291]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]': finished 2024-03-12T03:19:33.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:33 smithi094 ceph-mon[29291]: osdmap e262: 8 total, 8 up, 8 in 2024-03-12T03:19:33.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:33 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:33.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:33 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-33"}]': finished 2024-03-12T03:19:33.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:33 smithi151 ceph-mon[32923]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi094-78099-50"}]': finished 2024-03-12T03:19:33.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2339871669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:33.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:33 smithi151 ceph-mon[32923]: osdmap e261: 8 total, 8 up, 8 in 2024-03-12T03:19:33.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:33 smithi151 ceph-mon[32923]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]: dispatch 2024-03-12T03:19:33.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi094-78081-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi094-78081-41"}]': finished 2024-03-12T03:19:33.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:33 smithi151 ceph-mon[32923]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi094-78099-50"}]': finished 2024-03-12T03:19:33.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:33 smithi151 ceph-mon[32923]: osdmap e262: 8 total, 8 up, 8 in 2024-03-12T03:19:34.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[29291]: pgmap v350: 292 pgs: 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[29291]: osdmap e263: 8 total, 8 up, 8 in 2024-03-12T03:19:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4082184245' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[29291]: from='client.49141 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[40484]: pgmap v350: 292 pgs: 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[40484]: osdmap e263: 8 total, 8 up, 8 in 2024-03-12T03:19:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4082184245' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[40484]: from='client.49141 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:34 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:34 smithi151 ceph-mon[32923]: pgmap v350: 292 pgs: 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:34 smithi151 ceph-mon[32923]: osdmap e263: 8 total, 8 up, 8 in 2024-03-12T03:19:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4082184245' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:34 smithi151 ceph-mon[32923]: from='client.49141 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:34.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:34 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[40484]: pgmap v353: 332 pgs: 72 unknown, 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:19:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[40484]: from='client.49141 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[40484]: osdmap e264: 8 total, 8 up, 8 in 2024-03-12T03:19:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:36.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[29291]: pgmap v353: 332 pgs: 72 unknown, 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:19:36.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[29291]: from='client.49141 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:36.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:36.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[29291]: osdmap e264: 8 total, 8 up, 8 in 2024-03-12T03:19:36.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:36.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:36.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:36 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:36 smithi151 ceph-mon[32923]: pgmap v353: 332 pgs: 72 unknown, 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:19:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:36 smithi151 ceph-mon[32923]: from='client.49141 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi094-78099-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:36 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:36 smithi151 ceph-mon[32923]: osdmap e264: 8 total, 8 up, 8 in 2024-03-12T03:19:36.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:36.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:36.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:36 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:37.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi094-78081-41"}]': finished 2024-03-12T03:19:37.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:37.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: osdmap e265: 8 total, 8 up, 8 in 2024-03-12T03:19:37.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:37.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:37.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:37.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:37.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:37.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:37.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi094-78081-41"}]': finished 2024-03-12T03:19:37.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]': finished 2024-03-12T03:19:37.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:37.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: osdmap e266: 8 total, 8 up, 8 in 2024-03-12T03:19:37.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi094-78099-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:37.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi094-78081-41"}]': finished 2024-03-12T03:19:37.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:37.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: osdmap e265: 8 total, 8 up, 8 in 2024-03-12T03:19:37.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:37.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:37.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:37.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:37.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:37.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:37.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi094-78081-41"}]': finished 2024-03-12T03:19:37.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]': finished 2024-03-12T03:19:37.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:37.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: osdmap e266: 8 total, 8 up, 8 in 2024-03-12T03:19:37.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi094-78099-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:37.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi094-78081-41"}]': finished 2024-03-12T03:19:37.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: osdmap e265: 8 total, 8 up, 8 in 2024-03-12T03:19:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi094-78081-41"}]: dispatch 2024-03-12T03:19:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1819304615' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi094-78081-41"}]': finished 2024-03-12T03:19:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]': finished 2024-03-12T03:19:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: osdmap e266: 8 total, 8 up, 8 in 2024-03-12T03:19:37.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi094-78099-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:38.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:19:37 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:19:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[40484]: pgmap v356: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:19:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-35", "mode": "writeback"}]: dispatch 2024-03-12T03:19:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-35", "mode": "writeback"}]: dispatch 2024-03-12T03:19:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi094-78081-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-35", "mode": "writeback"}]': finished 2024-03-12T03:19:38.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[29291]: pgmap v356: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:19:38.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:38.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-35", "mode": "writeback"}]: dispatch 2024-03-12T03:19:38.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-35", "mode": "writeback"}]: dispatch 2024-03-12T03:19:38.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:38.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:38.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi094-78081-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:38.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:38.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:38 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-35", "mode": "writeback"}]': finished 2024-03-12T03:19:38.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:38 smithi151 ceph-mon[32923]: pgmap v356: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:19:38.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:38 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:38.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-35", "mode": "writeback"}]: dispatch 2024-03-12T03:19:38.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:38 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-35", "mode": "writeback"}]: dispatch 2024-03-12T03:19:38.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:38.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:38.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi094-78081-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:38.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:38 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:38.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:38 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-35", "mode": "writeback"}]': finished 2024-03-12T03:19:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi094-78081-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:39 smithi094 ceph-mon[29291]: osdmap e267: 8 total, 8 up, 8 in 2024-03-12T03:19:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi094-78081-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi094-78099-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]': finished 2024-03-12T03:19:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi094-78081-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:39 smithi094 ceph-mon[40484]: osdmap e267: 8 total, 8 up, 8 in 2024-03-12T03:19:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi094-78081-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi094-78099-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]': finished 2024-03-12T03:19:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi094-78081-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:39 smithi151 ceph-mon[32923]: osdmap e267: 8 total, 8 up, 8 in 2024-03-12T03:19:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi094-78081-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:39.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi094-78099-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]': finished 2024-03-12T03:19:40.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:19:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:19:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:19:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:40 smithi094 ceph-mon[29291]: pgmap v359: 292 pgs: 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:40 smithi094 ceph-mon[29291]: osdmap e268: 8 total, 8 up, 8 in 2024-03-12T03:19:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi094-78081-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi094-78081-42"}]': finished 2024-03-12T03:19:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:40 smithi094 ceph-mon[29291]: osdmap e269: 8 total, 8 up, 8 in 2024-03-12T03:19:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:40 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:40 smithi094 ceph-mon[40484]: pgmap v359: 292 pgs: 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:40 smithi094 ceph-mon[40484]: osdmap e268: 8 total, 8 up, 8 in 2024-03-12T03:19:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi094-78081-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi094-78081-42"}]': finished 2024-03-12T03:19:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:40 smithi094 ceph-mon[40484]: osdmap e269: 8 total, 8 up, 8 in 2024-03-12T03:19:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:40 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:40 smithi151 ceph-mon[32923]: pgmap v359: 292 pgs: 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:40 smithi151 ceph-mon[32923]: osdmap e268: 8 total, 8 up, 8 in 2024-03-12T03:19:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi094-78081-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi094-78081-42"}]': finished 2024-03-12T03:19:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:40 smithi151 ceph-mon[32923]: osdmap e269: 8 total, 8 up, 8 in 2024-03-12T03:19:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:40 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:42.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:42 smithi151 ceph-mon[32923]: pgmap v362: 308 pgs: 16 unknown, 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:42.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:42 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:42 smithi151 ceph-mon[32923]: osdmap e270: 8 total, 8 up, 8 in 2024-03-12T03:19:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:42.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:42 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:42.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:42 smithi094 ceph-mon[29291]: pgmap v362: 308 pgs: 16 unknown, 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:42 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:42 smithi094 ceph-mon[29291]: osdmap e270: 8 total, 8 up, 8 in 2024-03-12T03:19:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:42 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:42.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:42 smithi094 ceph-mon[40484]: pgmap v362: 308 pgs: 16 unknown, 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:42.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:42 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:42.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:42 smithi094 ceph-mon[40484]: osdmap e270: 8 total, 8 up, 8 in 2024-03-12T03:19:42.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:42.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:42.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:42 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:43.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]': finished 2024-03-12T03:19:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]': finished 2024-03-12T03:19:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[29291]: osdmap e271: 8 total, 8 up, 8 in 2024-03-12T03:19:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[29291]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi094-78099-52"}]': finished 2024-03-12T03:19:43.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]': finished 2024-03-12T03:19:43.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]': finished 2024-03-12T03:19:43.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[40484]: osdmap e271: 8 total, 8 up, 8 in 2024-03-12T03:19:43.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:43.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:43.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[40484]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:43.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi094-78099-52"}]': finished 2024-03-12T03:19:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi094-78099-52"}]': finished 2024-03-12T03:19:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-35"}]': finished 2024-03-12T03:19:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:43 smithi151 ceph-mon[32923]: osdmap e271: 8 total, 8 up, 8 in 2024-03-12T03:19:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi094-78099-52"}]: dispatch 2024-03-12T03:19:43.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:43.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:43 smithi151 ceph-mon[32923]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:43.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/949535722' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi094-78099-52"}]': finished 2024-03-12T03:19:44.218 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7036 ms) 2024-03-12T03:19:44.218 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-12T03:19:44.218 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6959 ms) 2024-03-12T03:19:44.218 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-12T03:19:44.218 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7007 ms) 2024-03-12T03:19:44.218 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-12T03:19:44.218 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6998 ms) 2024-03-12T03:19:44.219 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-12T03:19:44.219 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7029 ms) 2024-03-12T03:19:44.219 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-12T03:19:44.219 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7022 ms) 2024-03-12T03:19:44.219 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-12T03:19:44.219 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7027 ms) 2024-03-12T03:19:44.219 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-12T03:19:44.219 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6924 ms) 2024-03-12T03:19:44.219 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-12T03:19:44.219 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7018 ms) 2024-03-12T03:19:44.219 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-12T03:19:44.219 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7018 ms) 2024-03-12T03:19:44.219 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6970 ms) 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7017 ms) 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6990 ms) 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7020 ms) 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7016 ms) 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7015 ms) 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112066 ms total) 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: 2024-03-12T03:19:44.220 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [----------] Global test environment tear-down 2024-03-12T03:19:44.221 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (207545 ms total) 2024-03-12T03:19:44.221 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-12T03:19:44.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: pgmap v365: 292 pgs: 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 767 B/s wr, 1 op/s 2024-03-12T03:19:44.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi094-78081-42"}]': finished 2024-03-12T03:19:44.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: osdmap e272: 8 total, 8 up, 8 in 2024-03-12T03:19:44.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:44.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:44.617 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:44.617 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:44.617 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:44.617 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:44.617 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi094-78081-42"}]': finished 2024-03-12T03:19:44.617 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:44.617 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:44.617 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: osdmap e273: 8 total, 8 up, 8 in 2024-03-12T03:19:44.617 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi094-78099-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:44.617 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:44.617 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:44 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: pgmap v365: 292 pgs: 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 767 B/s wr, 1 op/s 2024-03-12T03:19:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi094-78081-42"}]': finished 2024-03-12T03:19:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: osdmap e272: 8 total, 8 up, 8 in 2024-03-12T03:19:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi094-78081-42"}]': finished 2024-03-12T03:19:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: osdmap e273: 8 total, 8 up, 8 in 2024-03-12T03:19:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi094-78099-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:44.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:44.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: pgmap v365: 292 pgs: 292 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 767 B/s wr, 1 op/s 2024-03-12T03:19:44.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi094-78081-42"}]': finished 2024-03-12T03:19:44.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: osdmap e272: 8 total, 8 up, 8 in 2024-03-12T03:19:44.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi094-78081-42"}]: dispatch 2024-03-12T03:19:44.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:44.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:44.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:44.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:44.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:44.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3544691003' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi094-78081-42"}]': finished 2024-03-12T03:19:44.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:44.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:44.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: osdmap e273: 8 total, 8 up, 8 in 2024-03-12T03:19:44.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi094-78099-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:44.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:44.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:44 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35"}]: dispatch 2024-03-12T03:19:45.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:45 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:45.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:45 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35"}]': finished 2024-03-12T03:19:45.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:45 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:45.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:45 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35"}]': finished 2024-03-12T03:19:45.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:45 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:45.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:45 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-35"}]': finished 2024-03-12T03:19:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:46 smithi151 ceph-mon[32923]: pgmap v368: 292 pgs: 292 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-12T03:19:46.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:46 smithi151 ceph-mon[32923]: osdmap e274: 8 total, 8 up, 8 in 2024-03-12T03:19:46.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi094-78099-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi094-78099-53"}]': finished 2024-03-12T03:19:46.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:46 smithi151 ceph-mon[32923]: osdmap e275: 8 total, 8 up, 8 in 2024-03-12T03:19:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:46 smithi094 ceph-mon[29291]: pgmap v368: 292 pgs: 292 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-12T03:19:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:46 smithi094 ceph-mon[29291]: osdmap e274: 8 total, 8 up, 8 in 2024-03-12T03:19:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi094-78099-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi094-78099-53"}]': finished 2024-03-12T03:19:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:46 smithi094 ceph-mon[29291]: osdmap e275: 8 total, 8 up, 8 in 2024-03-12T03:19:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:46 smithi094 ceph-mon[40484]: pgmap v368: 292 pgs: 292 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-12T03:19:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:46 smithi094 ceph-mon[40484]: osdmap e274: 8 total, 8 up, 8 in 2024-03-12T03:19:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi094-78099-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi094-78099-53"}]': finished 2024-03-12T03:19:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:46 smithi094 ceph-mon[40484]: osdmap e275: 8 total, 8 up, 8 in 2024-03-12T03:19:48.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:19:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:19:48.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:48 smithi094 ceph-mon[40484]: pgmap v371: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:48 smithi094 ceph-mon[40484]: osdmap e276: 8 total, 8 up, 8 in 2024-03-12T03:19:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:48 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:48 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:48 smithi094 ceph-mon[29291]: pgmap v371: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:48 smithi094 ceph-mon[29291]: osdmap e276: 8 total, 8 up, 8 in 2024-03-12T03:19:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:48 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:48 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:48 smithi151 ceph-mon[32923]: pgmap v371: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:48 smithi151 ceph-mon[32923]: osdmap e276: 8 total, 8 up, 8 in 2024-03-12T03:19:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:48 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:48 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[40484]: osdmap e277: 8 total, 8 up, 8 in 2024-03-12T03:19:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[29291]: osdmap e277: 8 total, 8 up, 8 in 2024-03-12T03:19:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:49 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:49 smithi151 ceph-mon[32923]: osdmap e277: 8 total, 8 up, 8 in 2024-03-12T03:19:49.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:49.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:49.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:49.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:49 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:50.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[40484]: pgmap v374: 292 pgs: 32 creating+peering, 260 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:19:50.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53"}]': finished 2024-03-12T03:19:50.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:50.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[40484]: osdmap e278: 8 total, 8 up, 8 in 2024-03-12T03:19:50.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-37"}]: dispatch 2024-03-12T03:19:50.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:50.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-37"}]: dispatch 2024-03-12T03:19:50.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[29291]: pgmap v374: 292 pgs: 32 creating+peering, 260 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:19:50.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53"}]': finished 2024-03-12T03:19:50.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:50.533 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[29291]: osdmap e278: 8 total, 8 up, 8 in 2024-03-12T03:19:50.533 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-37"}]: dispatch 2024-03-12T03:19:50.533 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:50.533 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-37"}]: dispatch 2024-03-12T03:19:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:50 smithi151 ceph-mon[32923]: pgmap v374: 292 pgs: 32 creating+peering, 260 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:19:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi094-78099-53"}]': finished 2024-03-12T03:19:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:50 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:50 smithi151 ceph-mon[32923]: osdmap e278: 8 total, 8 up, 8 in 2024-03-12T03:19:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-37"}]: dispatch 2024-03-12T03:19:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi094-78099-53"}]: dispatch 2024-03-12T03:19:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:50 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-37"}]: dispatch 2024-03-12T03:19:50.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:19:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:19:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:19:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi094-78099-53"}]': finished 2024-03-12T03:19:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-37"}]': finished 2024-03-12T03:19:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-37", "mode": "writeback"}]: dispatch 2024-03-12T03:19:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[40484]: osdmap e279: 8 total, 8 up, 8 in 2024-03-12T03:19:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-37", "mode": "writeback"}]: dispatch 2024-03-12T03:19:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[40484]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[40484]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi094-78099-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[40484]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi094-78099-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi094-78099-53"}]': finished 2024-03-12T03:19:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-37"}]': finished 2024-03-12T03:19:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-37", "mode": "writeback"}]: dispatch 2024-03-12T03:19:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[29291]: osdmap e279: 8 total, 8 up, 8 in 2024-03-12T03:19:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-37", "mode": "writeback"}]: dispatch 2024-03-12T03:19:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[29291]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[29291]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi094-78099-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:51.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:51 smithi094 ceph-mon[29291]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi094-78099-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3993496267' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi094-78099-53"}]': finished 2024-03-12T03:19:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:51 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-37"}]': finished 2024-03-12T03:19:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-37", "mode": "writeback"}]: dispatch 2024-03-12T03:19:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:51 smithi151 ceph-mon[32923]: osdmap e279: 8 total, 8 up, 8 in 2024-03-12T03:19:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:51 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-37", "mode": "writeback"}]: dispatch 2024-03-12T03:19:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:51 smithi151 ceph-mon[32923]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:51 smithi151 ceph-mon[32923]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi094-78099-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:51 smithi151 ceph-mon[32923]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi094-78099-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[29291]: pgmap v377: 292 pgs: 32 creating+peering, 260 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-37", "mode": "writeback"}]': finished 2024-03-12T03:19:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[29291]: from='client.56978 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi094-78099-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[29291]: osdmap e280: 8 total, 8 up, 8 in 2024-03-12T03:19:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi094-78099-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[29291]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi094-78099-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[40484]: pgmap v377: 292 pgs: 32 creating+peering, 260 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-37", "mode": "writeback"}]': finished 2024-03-12T03:19:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[40484]: from='client.56978 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi094-78099-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[40484]: osdmap e280: 8 total, 8 up, 8 in 2024-03-12T03:19:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi094-78099-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[40484]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi094-78099-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:52 smithi151 ceph-mon[32923]: pgmap v377: 292 pgs: 32 creating+peering, 260 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:19:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:52 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-37", "mode": "writeback"}]': finished 2024-03-12T03:19:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:52 smithi151 ceph-mon[32923]: from='client.56978 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi094-78099-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:52 smithi151 ceph-mon[32923]: osdmap e280: 8 total, 8 up, 8 in 2024-03-12T03:19:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi094-78099-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:52 smithi151 ceph-mon[32923]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi094-78099-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:19:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37"}]: dispatch 2024-03-12T03:19:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[29291]: osdmap e281: 8 total, 8 up, 8 in 2024-03-12T03:19:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37"}]: dispatch 2024-03-12T03:19:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[29291]: from='client.56978 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi094-78099-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi094-78099-54"}]': finished 2024-03-12T03:19:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37"}]': finished 2024-03-12T03:19:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[29291]: osdmap e282: 8 total, 8 up, 8 in 2024-03-12T03:19:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37"}]: dispatch 2024-03-12T03:19:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[40484]: osdmap e281: 8 total, 8 up, 8 in 2024-03-12T03:19:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37"}]: dispatch 2024-03-12T03:19:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[40484]: from='client.56978 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi094-78099-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi094-78099-54"}]': finished 2024-03-12T03:19:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37"}]': finished 2024-03-12T03:19:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:53 smithi094 ceph-mon[40484]: osdmap e282: 8 total, 8 up, 8 in 2024-03-12T03:19:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:53 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:19:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37"}]: dispatch 2024-03-12T03:19:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:53 smithi151 ceph-mon[32923]: osdmap e281: 8 total, 8 up, 8 in 2024-03-12T03:19:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:53 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37"}]: dispatch 2024-03-12T03:19:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:53 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:19:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:53 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:19:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:53 smithi151 ceph-mon[32923]: from='client.56978 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi094-78099-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi094-78099-54"}]': finished 2024-03-12T03:19:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:53 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-37"}]': finished 2024-03-12T03:19:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:53 smithi151 ceph-mon[32923]: osdmap e282: 8 total, 8 up, 8 in 2024-03-12T03:19:54.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:54 smithi094 ceph-mon[40484]: pgmap v380: 292 pgs: 27 creating+peering, 265 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:54.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:54 smithi094 ceph-mon[40484]: osdmap e283: 8 total, 8 up, 8 in 2024-03-12T03:19:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:54 smithi094 ceph-mon[29291]: pgmap v380: 292 pgs: 27 creating+peering, 265 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:54 smithi094 ceph-mon[29291]: osdmap e283: 8 total, 8 up, 8 in 2024-03-12T03:19:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:54 smithi151 ceph-mon[32923]: pgmap v380: 292 pgs: 27 creating+peering, 265 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:19:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:54 smithi151 ceph-mon[32923]: osdmap e283: 8 total, 8 up, 8 in 2024-03-12T03:19:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:56 smithi094 ceph-mon[40484]: pgmap v383: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:19:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:56 smithi094 ceph-mon[40484]: osdmap e284: 8 total, 8 up, 8 in 2024-03-12T03:19:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:56 smithi094 ceph-mon[40484]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:56 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:56 smithi094 ceph-mon[29291]: pgmap v383: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:19:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:56 smithi094 ceph-mon[29291]: osdmap e284: 8 total, 8 up, 8 in 2024-03-12T03:19:56.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:56 smithi094 ceph-mon[29291]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:56.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:56 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:56 smithi151 ceph-mon[32923]: pgmap v383: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:19:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:56 smithi151 ceph-mon[32923]: osdmap e284: 8 total, 8 up, 8 in 2024-03-12T03:19:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:56 smithi151 ceph-mon[32923]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:56.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:56 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:19:57.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[29291]: from='client.56978 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]': finished 2024-03-12T03:19:57.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:57.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[29291]: osdmap e285: 8 total, 8 up, 8 in 2024-03-12T03:19:57.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:57.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:57.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:57.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[29291]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:57.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:57.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[29291]: from='client.56978 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]': finished 2024-03-12T03:19:57.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[29291]: osdmap e286: 8 total, 8 up, 8 in 2024-03-12T03:19:57.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[40484]: from='client.56978 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]': finished 2024-03-12T03:19:57.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:57.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[40484]: osdmap e285: 8 total, 8 up, 8 in 2024-03-12T03:19:57.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:57.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:57.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:57.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[40484]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:57.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:57.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[40484]: from='client.56978 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]': finished 2024-03-12T03:19:57.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:57 smithi094 ceph-mon[40484]: osdmap e286: 8 total, 8 up, 8 in 2024-03-12T03:19:57.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:57 smithi151 ceph-mon[32923]: from='client.56978 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi094-78099-54"}]': finished 2024-03-12T03:19:57.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:57 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:19:57.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:57 smithi151 ceph-mon[32923]: osdmap e285: 8 total, 8 up, 8 in 2024-03-12T03:19:57.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:57.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4072025841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:57.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:57 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:19:57.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:57 smithi151 ceph-mon[32923]: from='client.56978 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]: dispatch 2024-03-12T03:19:57.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:57 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:19:57.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:57 smithi151 ceph-mon[32923]: from='client.56978 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi094-78099-54"}]': finished 2024-03-12T03:19:57.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:57 smithi151 ceph-mon[32923]: osdmap e286: 8 total, 8 up, 8 in 2024-03-12T03:19:58.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:19:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:19:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: pgmap v386: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:19:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-39"}]: dispatch 2024-03-12T03:19:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-39"}]: dispatch 2024-03-12T03:19:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi094-78099-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:58.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi094-78099-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:58.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-39"}]': finished 2024-03-12T03:19:58.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi094-78099-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:58.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: osdmap e287: 8 total, 8 up, 8 in 2024-03-12T03:19:58.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-39", "mode": "writeback"}]: dispatch 2024-03-12T03:19:58.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi094-78099-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-39", "mode": "writeback"}]: dispatch 2024-03-12T03:19:58.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[29291]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi094-78099-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: pgmap v386: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:19:58.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-39"}]: dispatch 2024-03-12T03:19:58.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-39"}]: dispatch 2024-03-12T03:19:58.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi094-78099-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:58.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi094-78099-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:58.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-39"}]': finished 2024-03-12T03:19:58.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi094-78099-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:58.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: osdmap e287: 8 total, 8 up, 8 in 2024-03-12T03:19:58.462 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-39", "mode": "writeback"}]: dispatch 2024-03-12T03:19:58.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi094-78099-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-39", "mode": "writeback"}]: dispatch 2024-03-12T03:19:58.463 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:58 smithi094 ceph-mon[40484]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi094-78099-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: pgmap v386: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:19:58.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-39"}]: dispatch 2024-03-12T03:19:58.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-39"}]: dispatch 2024-03-12T03:19:58.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi094-78099-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi094-78099-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:19:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-39"}]': finished 2024-03-12T03:19:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi094-78099-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:19:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: osdmap e287: 8 total, 8 up, 8 in 2024-03-12T03:19:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-39", "mode": "writeback"}]: dispatch 2024-03-12T03:19:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi094-78099-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-39", "mode": "writeback"}]: dispatch 2024-03-12T03:19:58.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:58 smithi151 ceph-mon[32923]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi094-78099-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:19:59.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:59 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:19:59 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-39", "mode": "writeback"}]': finished 2024-03-12T03:19:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:59 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:19:59 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-39", "mode": "writeback"}]': finished 2024-03-12T03:19:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:19:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:59 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:19:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:19:59 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-39", "mode": "writeback"}]': finished 2024-03-12T03:20:00.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:20:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:20:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: pgmap v389: 292 pgs: 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: osdmap e288: 8 total, 8 up, 8 in 2024-03-12T03:20:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 3 pool(s) do not have an application enabled 2024-03-12T03:20:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-12T03:20:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: pool 'test-rados-api-smithi094-78452-39' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-12T03:20:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-12T03:20:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-12T03:20:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: application not enabled on pool 'ListObjectssmithi094-79253-1' 2024-03-12T03:20:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: application not enabled on pool 'SetOpFlagssmithi094-79449-1' 2024-03-12T03:20:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-12T03:20:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi094-78099-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi094-78099-55"}]': finished 2024-03-12T03:20:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:20:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: osdmap e289: 8 total, 8 up, 8 in 2024-03-12T03:20:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39"}]: dispatch 2024-03-12T03:20:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39"}]: dispatch 2024-03-12T03:20:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: pgmap v389: 292 pgs: 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: osdmap e288: 8 total, 8 up, 8 in 2024-03-12T03:20:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:00.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 3 pool(s) do not have an application enabled 2024-03-12T03:20:00.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-12T03:20:00.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: pool 'test-rados-api-smithi094-78452-39' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-12T03:20:00.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-12T03:20:00.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-12T03:20:00.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: application not enabled on pool 'ListObjectssmithi094-79253-1' 2024-03-12T03:20:00.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: application not enabled on pool 'SetOpFlagssmithi094-79449-1' 2024-03-12T03:20:00.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-12T03:20:00.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi094-78099-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi094-78099-55"}]': finished 2024-03-12T03:20:00.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:20:00.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: osdmap e289: 8 total, 8 up, 8 in 2024-03-12T03:20:00.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39"}]: dispatch 2024-03-12T03:20:00.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:00 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39"}]: dispatch 2024-03-12T03:20:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: pgmap v389: 292 pgs: 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: osdmap e288: 8 total, 8 up, 8 in 2024-03-12T03:20:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 3 pool(s) do not have an application enabled 2024-03-12T03:20:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-12T03:20:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: pool 'test-rados-api-smithi094-78452-39' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-12T03:20:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-12T03:20:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-12T03:20:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: application not enabled on pool 'ListObjectssmithi094-79253-1' 2024-03-12T03:20:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: application not enabled on pool 'SetOpFlagssmithi094-79449-1' 2024-03-12T03:20:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-12T03:20:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi094-78099-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi094-78099-55"}]': finished 2024-03-12T03:20:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:20:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: osdmap e289: 8 total, 8 up, 8 in 2024-03-12T03:20:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39"}]: dispatch 2024-03-12T03:20:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:00 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39"}]: dispatch 2024-03-12T03:20:01.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:01 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:20:01.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:01 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39"}]': finished 2024-03-12T03:20:01.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:01 smithi094 ceph-mon[40484]: osdmap e290: 8 total, 8 up, 8 in 2024-03-12T03:20:01.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:01 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:20:01.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:01 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39"}]': finished 2024-03-12T03:20:01.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:01 smithi094 ceph-mon[29291]: osdmap e290: 8 total, 8 up, 8 in 2024-03-12T03:20:01.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:01 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:20:01.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:01 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-39"}]': finished 2024-03-12T03:20:01.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:01 smithi151 ceph-mon[32923]: osdmap e290: 8 total, 8 up, 8 in 2024-03-12T03:20:02.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-12T03:20:02.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-12T03:20:02.182 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: seed 78452 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (318 ms) 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (17175 ms) 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (35 ms) 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (17528 ms total) 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (8208 ms) 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-12T03:20:02.183 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (9866 ms) 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9995 ms) 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: my_snaps [3] 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: my_snaps [4,3] 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: my_snaps [5,4,3] 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: promoting some heads 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: promoting from clones for snap 3 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: waiting for scrubs... 2024-03-12T03:20:02.184 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: done waiting 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46316 ms) 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9710 ms) 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7994 ms) 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8033 ms) 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8028 ms) 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10040 ms) 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8923 ms) 2024-03-12T03:20:02.185 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-12T03:20:02.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10261 ms) 2024-03-12T03:20:02.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-12T03:20:02.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12745 ms) 2024-03-12T03:20:02.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-12T03:20:02.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8227 ms) 2024-03-12T03:20:02.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-12T03:20:02.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (7787 ms) 2024-03-12T03:20:02.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-12T03:20:02.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13028 ms) 2024-03-12T03:20:02.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-12T03:20:02.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7951 ms) 2024-03-12T03:20:02.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-12T03:20:02.186 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8015 ms) 2024-03-12T03:20:02.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:02 smithi094 ceph-mon[40484]: pgmap v392: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:02.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:02 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:02.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:02 smithi094 ceph-mon[40484]: osdmap e291: 8 total, 8 up, 8 in 2024-03-12T03:20:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:20:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:02 smithi094 ceph-mon[40484]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:20:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:02 smithi094 ceph-mon[29291]: pgmap v392: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:02 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:02 smithi094 ceph-mon[29291]: osdmap e291: 8 total, 8 up, 8 in 2024-03-12T03:20:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:20:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:02 smithi094 ceph-mon[29291]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:20:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:02 smithi151 ceph-mon[32923]: pgmap v392: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:02 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:02 smithi151 ceph-mon[32923]: osdmap e291: 8 total, 8 up, 8 in 2024-03-12T03:20:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:20:02.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:02 smithi151 ceph-mon[32923]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:20:03.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[29291]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]': finished 2024-03-12T03:20:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[29291]: osdmap e292: 8 total, 8 up, 8 in 2024-03-12T03:20:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:20:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[29291]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:20:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[40484]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]': finished 2024-03-12T03:20:03.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[40484]: osdmap e292: 8 total, 8 up, 8 in 2024-03-12T03:20:03.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:20:03.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[40484]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:20:03.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:03.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:03 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:03 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:03 smithi151 ceph-mon[32923]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi094-78099-55"}]': finished 2024-03-12T03:20:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:03 smithi151 ceph-mon[32923]: osdmap e292: 8 total, 8 up, 8 in 2024-03-12T03:20:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/419704730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:20:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:03 smithi151 ceph-mon[32923]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]: dispatch 2024-03-12T03:20:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:03 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: pgmap v395: 260 pgs: 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:20:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:20:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]': finished 2024-03-12T03:20:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: osdmap e293: 8 total, 8 up, 8 in 2024-03-12T03:20:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi094-78099-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:04.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:20:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: pgmap v395: 260 pgs: 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:20:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:20:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]': finished 2024-03-12T03:20:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:04.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: osdmap e293: 8 total, 8 up, 8 in 2024-03-12T03:20:04.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:04.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:04.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:04.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:04.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi094-78099-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:04.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:04.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:04.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:04 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:20:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: pgmap v395: 260 pgs: 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:20:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:20:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi094-78099-55"}]': finished 2024-03-12T03:20:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: osdmap e293: 8 total, 8 up, 8 in 2024-03-12T03:20:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi094-78099-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:04 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:20:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[29291]: pgmap v398: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-12T03:20:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi094-78099-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[29291]: osdmap e294: 8 total, 8 up, 8 in 2024-03-12T03:20:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi094-78099-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:20:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:20:06.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[40484]: pgmap v398: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-12T03:20:06.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:06.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi094-78099-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:06.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[40484]: osdmap e294: 8 total, 8 up, 8 in 2024-03-12T03:20:06.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi094-78099-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:06.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:20:06.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:06 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:20:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:06 smithi151 ceph-mon[32923]: pgmap v398: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-12T03:20:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:06 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi094-78099-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:06 smithi151 ceph-mon[32923]: osdmap e294: 8 total, 8 up, 8 in 2024-03-12T03:20:06.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi094-78099-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:06.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:20:06.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:06 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:20:07.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:20:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[29291]: osdmap e295: 8 total, 8 up, 8 in 2024-03-12T03:20:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi094-78099-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi094-78099-56"}]': finished 2024-03-12T03:20:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:20:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[29291]: osdmap e296: 8 total, 8 up, 8 in 2024-03-12T03:20:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:20:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:20:07.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:20:07.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:07.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[40484]: osdmap e295: 8 total, 8 up, 8 in 2024-03-12T03:20:07.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:07.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi094-78099-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi094-78099-56"}]': finished 2024-03-12T03:20:07.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:20:07.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[40484]: osdmap e296: 8 total, 8 up, 8 in 2024-03-12T03:20:07.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:20:07.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:07 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:20:07.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:07 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:20:07.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:07.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:07 smithi151 ceph-mon[32923]: osdmap e295: 8 total, 8 up, 8 in 2024-03-12T03:20:07.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:07 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:07.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi094-78099-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi094-78099-56"}]': finished 2024-03-12T03:20:07.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:07 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:20:07.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:07 smithi151 ceph-mon[32923]: osdmap e296: 8 total, 8 up, 8 in 2024-03-12T03:20:07.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:20:07.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:07 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:20:08.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:20:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:20:08.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:08 smithi094 ceph-mon[29291]: pgmap v401: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-12T03:20:08.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:08 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:08.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:08 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-12T03:20:08.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:08 smithi094 ceph-mon[29291]: osdmap e297: 8 total, 8 up, 8 in 2024-03-12T03:20:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:08 smithi094 ceph-mon[40484]: pgmap v401: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-12T03:20:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:08 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:08 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-12T03:20:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:08 smithi094 ceph-mon[40484]: osdmap e297: 8 total, 8 up, 8 in 2024-03-12T03:20:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:08 smithi151 ceph-mon[32923]: pgmap v401: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-12T03:20:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:08 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:08 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-12T03:20:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:08 smithi151 ceph-mon[32923]: osdmap e297: 8 total, 8 up, 8 in 2024-03-12T03:20:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[29291]: osdmap e298: 8 total, 8 up, 8 in 2024-03-12T03:20:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41"}]: dispatch 2024-03-12T03:20:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41"}]: dispatch 2024-03-12T03:20:09.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:09.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[40484]: osdmap e298: 8 total, 8 up, 8 in 2024-03-12T03:20:09.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:09.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:09.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:09.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41"}]: dispatch 2024-03-12T03:20:09.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:09 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41"}]: dispatch 2024-03-12T03:20:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:09 smithi151 ceph-mon[32923]: osdmap e298: 8 total, 8 up, 8 in 2024-03-12T03:20:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:09 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41"}]: dispatch 2024-03-12T03:20:09.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:09 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41"}]: dispatch 2024-03-12T03:20:10.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:10 smithi094 ceph-mon[40484]: pgmap v404: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi094-78099-56"}]': finished 2024-03-12T03:20:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:10 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41"}]': finished 2024-03-12T03:20:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:10 smithi094 ceph-mon[40484]: osdmap e299: 8 total, 8 up, 8 in 2024-03-12T03:20:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:10 smithi094 ceph-mon[29291]: pgmap v404: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi094-78099-56"}]': finished 2024-03-12T03:20:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:10 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41"}]': finished 2024-03-12T03:20:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:10 smithi094 ceph-mon[29291]: osdmap e299: 8 total, 8 up, 8 in 2024-03-12T03:20:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:10.711 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:20:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:20:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:20:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:10 smithi151 ceph-mon[32923]: pgmap v404: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi094-78099-56"}]': finished 2024-03-12T03:20:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:10 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-41"}]': finished 2024-03-12T03:20:10.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:10 smithi151 ceph-mon[32923]: osdmap e299: 8 total, 8 up, 8 in 2024-03-12T03:20:10.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi094-78099-56"}]: dispatch 2024-03-12T03:20:12.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[29291]: pgmap v407: 292 pgs: 292 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi094-78099-56"}]': finished 2024-03-12T03:20:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[29291]: osdmap e300: 8 total, 8 up, 8 in 2024-03-12T03:20:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[29291]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[29291]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi094-78099-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:12.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[29291]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi094-78099-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:12.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[40484]: pgmap v407: 292 pgs: 292 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:12.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi094-78099-56"}]': finished 2024-03-12T03:20:12.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[40484]: osdmap e300: 8 total, 8 up, 8 in 2024-03-12T03:20:12.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:12.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[40484]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:12.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:12.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[40484]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:12.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi094-78099-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:12.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:12 smithi094 ceph-mon[40484]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi094-78099-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:12.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:12 smithi151 ceph-mon[32923]: pgmap v407: 292 pgs: 292 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:12.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3738624032' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi094-78099-56"}]': finished 2024-03-12T03:20:12.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:12 smithi151 ceph-mon[32923]: osdmap e300: 8 total, 8 up, 8 in 2024-03-12T03:20:12.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:12.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:12 smithi151 ceph-mon[32923]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:12.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:12.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:12 smithi151 ceph-mon[32923]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:12.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi094-78099-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:12.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:12 smithi151 ceph-mon[32923]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi094-78099-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:13 smithi094 ceph-mon[40484]: from='client.56996 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi094-78099-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:13 smithi094 ceph-mon[40484]: osdmap e301: 8 total, 8 up, 8 in 2024-03-12T03:20:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi094-78099-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:13 smithi094 ceph-mon[40484]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi094-78099-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:13.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:13 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:13.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:13 smithi094 ceph-mon[29291]: from='client.56996 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi094-78099-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:13.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:13 smithi094 ceph-mon[29291]: osdmap e301: 8 total, 8 up, 8 in 2024-03-12T03:20:13.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi094-78099-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:13.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:13.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:13 smithi094 ceph-mon[29291]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi094-78099-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:13.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:13 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:13.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:13 smithi151 ceph-mon[32923]: from='client.56996 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi094-78099-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:13.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:13 smithi151 ceph-mon[32923]: osdmap e301: 8 total, 8 up, 8 in 2024-03-12T03:20:13.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi094-78099-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:13.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:13.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:13 smithi151 ceph-mon[32923]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi094-78099-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:13.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:13 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[29291]: pgmap v410: 292 pgs: 5 creating+peering, 27 unknown, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi094-78452-6","var": "pg_num","format": "json"}]: dispatch 2024-03-12T03:20:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:14.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[29291]: osdmap e302: 8 total, 8 up, 8 in 2024-03-12T03:20:14.461 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:14.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[40484]: pgmap v410: 292 pgs: 5 creating+peering, 27 unknown, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:14.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:14.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:14.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi094-78452-6","var": "pg_num","format": "json"}]: dispatch 2024-03-12T03:20:14.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:14.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[40484]: osdmap e302: 8 total, 8 up, 8 in 2024-03-12T03:20:14.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:14 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:14.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:14 smithi151 ceph-mon[32923]: pgmap v410: 292 pgs: 5 creating+peering, 27 unknown, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:14.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:14 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:14.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:14 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:14.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi094-78452-6","var": "pg_num","format": "json"}]: dispatch 2024-03-12T03:20:14.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:14 smithi151 ceph-mon[32923]: osdmap e302: 8 total, 8 up, 8 in 2024-03-12T03:20:14.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:14 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:15.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:15 smithi151 ceph-mon[32923]: from='client.56996 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi094-78099-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi094-78099-57"}]': finished 2024-03-12T03:20:15.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:15 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:15.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:15 smithi151 ceph-mon[32923]: osdmap e303: 8 total, 8 up, 8 in 2024-03-12T03:20:15.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:15 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-12T03:20:15.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:15 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-12T03:20:15.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:15 smithi094 ceph-mon[29291]: from='client.56996 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi094-78099-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi094-78099-57"}]': finished 2024-03-12T03:20:15.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:15 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:15.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:15 smithi094 ceph-mon[29291]: osdmap e303: 8 total, 8 up, 8 in 2024-03-12T03:20:15.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:15 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-12T03:20:15.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:15 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-12T03:20:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:15 smithi094 ceph-mon[40484]: from='client.56996 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi094-78099-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi094-78099-57"}]': finished 2024-03-12T03:20:15.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:15 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:15.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:15 smithi094 ceph-mon[40484]: osdmap e303: 8 total, 8 up, 8 in 2024-03-12T03:20:15.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:15 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-12T03:20:15.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:15 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-12T03:20:16.611 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:16 smithi094 ceph-mon[40484]: pgmap v413: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:16.611 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:16 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_count","val": "8"}]': finished 2024-03-12T03:20:16.611 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:16.611 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:16 smithi094 ceph-mon[40484]: osdmap e304: 8 total, 8 up, 8 in 2024-03-12T03:20:16.611 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:16 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:16.612 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:16 smithi094 ceph-mon[29291]: pgmap v413: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:16.612 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:16 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_count","val": "8"}]': finished 2024-03-12T03:20:16.612 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:16.612 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:16 smithi094 ceph-mon[29291]: osdmap e304: 8 total, 8 up, 8 in 2024-03-12T03:20:16.612 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:16 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:16.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:16 smithi151 ceph-mon[32923]: pgmap v413: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:16.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:16 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_count","val": "8"}]': finished 2024-03-12T03:20:16.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:16.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:16 smithi151 ceph-mon[32923]: osdmap e304: 8 total, 8 up, 8 in 2024-03-12T03:20:16.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:16 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:17.457 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-03-12T03:20:17.458 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-12T03:20:17.458 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: ok, hit_set contains 260:602f83fe:::foo:head 2024-03-12T03:20:17.458 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9016 ms) 2024-03-12T03:20:17.458 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-12T03:20:17.458 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg_num = 32 2024-03-12T03:20:17.458 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 0 ls 1710213618,0 2024-03-12T03:20:17.458 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 1 ls 1710213618,0 2024-03-12T03:20:17.458 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 2 ls 1710213618,0 2024-03-12T03:20:17.458 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 3 ls 1710213618,0 2024-03-12T03:20:17.458 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 4 ls 1710213618,0 2024-03-12T03:20:17.458 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 5 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 6 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 7 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 8 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 9 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 10 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 11 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 12 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 13 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 14 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 15 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 16 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 17 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 18 ls 1710213618,0 2024-03-12T03:20:17.459 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 19 ls 1710213618,0 2024-03-12T03:20:17.460 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 20 ls 1710213618,0 2024-03-12T03:20:17.460 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 21 ls 1710213618,0 2024-03-12T03:20:17.460 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 22 ls 1710213618,0 2024-03-12T03:20:17.460 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 23 ls 1710213618,0 2024-03-12T03:20:17.460 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 24 ls 1710213618,0 2024-03-12T03:20:17.460 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 25 ls 1710213618,0 2024-03-12T03:20:17.460 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 26 ls 1710213618,0 2024-03-12T03:20:17.460 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 27 ls 1710213618,0 2024-03-12T03:20:17.460 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 28 ls 1710213618,0 2024-03-12T03:20:17.460 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 29 ls 1710213618,0 2024-03-12T03:20:17.460 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 30 ls 1710213618,0 2024-03-12T03:20:17.460 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg 31 ls 1710213618,0 2024-03-12T03:20:17.460 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: pg_num = 32 2024-03-12T03:20:17.461 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:6cac518f:::0:head 2024-03-12T03:20:17.461 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:02547ec2:::1:head 2024-03-12T03:20:17.461 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:f905c69b:::2:head 2024-03-12T03:20:17.461 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:cfc208b3:::3:head 2024-03-12T03:20:17.461 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:d83876eb:::4:head 2024-03-12T03:20:17.461 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:b29083e3:::5:head 2024-03-12T03:20:17.461 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:c4fdafeb:::6:head 2024-03-12T03:20:17.461 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:5c6b0b28:::7:head 2024-03-12T03:20:17.461 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:bd63b0f1:::8:head 2024-03-12T03:20:17.461 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:e960b815:::9:head 2024-03-12T03:20:17.461 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:52ea6a34:::10:head 2024-03-12T03:20:17.461 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:89d3ae78:::11:head 2024-03-12T03:20:17.462 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:de5d7c5f:::12:head 2024-03-12T03:20:17.462 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:566253c9:::13:head 2024-03-12T03:20:17.462 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:62a1935d:::14:head 2024-03-12T03:20:17.462 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:863748b0:::15:head 2024-03-12T03:20:17.462 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:3958e169:::16:head 2024-03-12T03:20:17.462 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:4d4dabf9:::17:head 2024-03-12T03:20:17.462 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:8391935d:::18:head 2024-03-12T03:20:17.462 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:28883081:::19:head 2024-03-12T03:20:17.462 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:69259c59:::20:head 2024-03-12T03:20:17.462 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:4bdb80b7:::21:head 2024-03-12T03:20:17.462 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:a11c5d71:::22:head 2024-03-12T03:20:17.462 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:271af37b:::23:head 2024-03-12T03:20:17.462 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:95b121be:::24:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:58d1031b:::25:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:0a050783:::26:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:c709704c:::27:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:cbe56eaf:::28:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:86b4b162:::29:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:70d89383:::30:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:dd450c7c:::31:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:6d5729b1:::32:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:c388f3fb:::33:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:56cfea31:::34:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:9dbc1bf7:::35:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:40b74ccd:::36:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:4d5aaf42:::37:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:920f362c:::38:head 2024-03-12T03:20:17.463 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:6cc53222:::39:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:9cad833f:::40:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:1ea84d41:::41:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:c4480ef6:::42:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:a694361e:::43:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:d1bd33e9:::44:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:ddc2cd5d:::45:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:2b782207:::46:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:7b187fca:::47:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:90ecdf6f:::48:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:a5ed95fe:::49:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:ea0eaa55:::50:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:f33ef17b:::51:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:a0d1b2f6:::52:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:60c5229e:::53:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:edcbc575:::54:head 2024-03-12T03:20:17.464 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:102cf253:::55:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:efb7fb0b:::56:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:50d0a326:::57:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:d4dc5daf:::58:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:3a130462:::59:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:ec87ed71:::60:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:d5bc9454:::61:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:3ddfe313:::62:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:7c2816b9:::63:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:47e00e4d:::64:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:c6410c18:::65:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:b48ed237:::66:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:cd63ad31:::67:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:b179e92b:::68:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:0d9f741a:::69:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:6d3352ae:::70:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:c6d5c19e:::71:head 2024-03-12T03:20:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:bc4729c3:::72:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:77e930b9:::73:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:0abeecfd:::74:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:b7c37e15:::75:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:b6378398:::76:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:02bd68de:::77:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:cc795d2d:::78:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:630d4fea:::79:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:e0d29ef5:::80:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:fd6f13d2:::81:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:606461d5:::82:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:eadbdc43:::83:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:8761d0bb:::84:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:9ef0186f:::85:head 2024-03-12T03:20:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:e0d41294:::86:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:961de695:::87:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:1423148f:::88:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:633a8fa2:::89:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:a8653809:::90:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:3dac8b33:::91:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:35aad435:::92:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:f6dcc343:::93:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:dbbdad87:::94:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:1cb48ce0:::95:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:03cd461c:::96:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:17a4ea99:::97:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:9993c9a7:::98:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:6394211c:::99:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:94c7ae57:::100:head 2024-03-12T03:20:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:6fdee5bb:::101:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:9a477fd1:::102:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:eb850916:::103:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:affc56b9:::104:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:b42dc814:::105:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:f319f8f0:::106:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:9a40b9de:::107:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:8b524f28:::108:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:e3de589f:::109:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:90f90a5b:::110:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:a7b4f1d7:::111:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:af51766e:::112:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:b6f90bd1:::113:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:e0261208:::114:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:c9569ef7:::115:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:61bebe50:::116:head 2024-03-12T03:20:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:fe93412b:::117:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:d3d38bee:::118:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:3100ba0c:::119:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:d0560ada:::120:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:f0ea8b35:::121:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:766f231a:::122:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:a07a2582:::123:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:bd7c6b3a:::124:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:fb2ddaff:::125:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:4408e1fe:::126:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:ee1df7a7:::127:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:c3002909:::128:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:4f48ffa9:::129:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:edf38733:::130:head 2024-03-12T03:20:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:c08425c0:::131:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:5f902d98:::132:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:41ea2c93:::133:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:813cee13:::134:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:0131818d:::135:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:26ba5a85:::136:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:381b8a5a:::137:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:28797e47:::138:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:bfca7f22:::139:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:36807075:::140:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:80b03975:::141:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:5c15709b:::142:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:f39ea15e:::143:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:ea992956:::144:head 2024-03-12T03:20:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:48887b1c:::145:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:9f24a9dd:::146:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:987f100b:::147:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:d2dd3581:::148:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:7fed1808:::149:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:c80b70e9:::150:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:85ed90f9:::151:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:36428b24:::152:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:d044c34a:::153:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:7c18bf58:::154:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:d1c21232:::155:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:a7a3c575:::156:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:87da0633:::157:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:d5ac3822:::158:head 2024-03-12T03:20:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:3f20522d:::159:head 2024-03-12T03:20:17.472 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:6ca26563:::160:head 2024-03-12T03:20:17.472 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:532ce135:::161:head 2024-03-12T03:20:17.472 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:c78863e6:::162:head 2024-03-12T03:20:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:17 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:20:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:17 smithi094 ceph-mon[29291]: osdmap e305: 8 total, 8 up, 8 in 2024-03-12T03:20:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:17 smithi094 ceph-mon[29291]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-12T03:20:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:17 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-12T03:20:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:17 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:20:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:17 smithi094 ceph-mon[40484]: osdmap e305: 8 total, 8 up, 8 in 2024-03-12T03:20:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:17 smithi094 ceph-mon[40484]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-12T03:20:17.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:17 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-12T03:20:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:17 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:20:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:17 smithi151 ceph-mon[32923]: osdmap e305: 8 total, 8 up, 8 in 2024-03-12T03:20:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:17.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:17 smithi151 ceph-mon[32923]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:17.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-12T03:20:17.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:17 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-12T03:20:18.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:20:17 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:20:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[40484]: pgmap v416: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[40484]: from='client.56996 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]': finished 2024-03-12T03:20:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-12T03:20:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[40484]: osdmap e306: 8 total, 8 up, 8 in 2024-03-12T03:20:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[40484]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi094-78452-43","var": "pg_num","format": "json"}]: dispatch 2024-03-12T03:20:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43"}]: dispatch 2024-03-12T03:20:18.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43"}]: dispatch 2024-03-12T03:20:18.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[29291]: pgmap v416: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:18.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[29291]: from='client.56996 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]': finished 2024-03-12T03:20:18.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-12T03:20:18.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[29291]: osdmap e306: 8 total, 8 up, 8 in 2024-03-12T03:20:18.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:18.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[29291]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:18.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi094-78452-43","var": "pg_num","format": "json"}]: dispatch 2024-03-12T03:20:18.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:18.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:18.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43"}]: dispatch 2024-03-12T03:20:18.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:18 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43"}]: dispatch 2024-03-12T03:20:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:18 smithi151 ceph-mon[32923]: pgmap v416: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:18 smithi151 ceph-mon[32923]: from='client.56996 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi094-78099-57"}]': finished 2024-03-12T03:20:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:18 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-12T03:20:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:18 smithi151 ceph-mon[32923]: osdmap e306: 8 total, 8 up, 8 in 2024-03-12T03:20:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2485771748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:18 smithi151 ceph-mon[32923]: from='client.56996 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]: dispatch 2024-03-12T03:20:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi094-78452-43","var": "pg_num","format": "json"}]: dispatch 2024-03-12T03:20:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:18 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43"}]: dispatch 2024-03-12T03:20:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:18 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43"}]: dispatch 2024-03-12T03:20:19.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[40484]: from='client.56996 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]': finished 2024-03-12T03:20:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43"}]': finished 2024-03-12T03:20:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[40484]: osdmap e307: 8 total, 8 up, 8 in 2024-03-12T03:20:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[40484]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[40484]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi094-78099-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[40484]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi094-78099-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:19.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:19.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[29291]: from='client.56996 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]': finished 2024-03-12T03:20:19.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43"}]': finished 2024-03-12T03:20:19.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[29291]: osdmap e307: 8 total, 8 up, 8 in 2024-03-12T03:20:19.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:19.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[29291]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:19.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:19.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[29291]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:19.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi094-78099-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:19.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[29291]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi094-78099-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:19.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:19 smithi151 ceph-mon[32923]: from='client.56996 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi094-78099-57"}]': finished 2024-03-12T03:20:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:19 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-43"}]': finished 2024-03-12T03:20:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:19 smithi151 ceph-mon[32923]: osdmap e307: 8 total, 8 up, 8 in 2024-03-12T03:20:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:19 smithi151 ceph-mon[32923]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:19 smithi151 ceph-mon[32923]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi094-78099-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:19 smithi151 ceph-mon[32923]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi094-78099-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:20.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:20 smithi094 ceph-mon[40484]: pgmap v419: 292 pgs: 292 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-12T03:20:20.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:20 smithi094 ceph-mon[40484]: from='client.57002 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi094-78099-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:20.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:20 smithi094 ceph-mon[40484]: osdmap e308: 8 total, 8 up, 8 in 2024-03-12T03:20:20.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi094-78099-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:20.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:20 smithi094 ceph-mon[40484]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi094-78099-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:20.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:20 smithi094 ceph-mon[29291]: pgmap v419: 292 pgs: 292 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-12T03:20:20.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:20 smithi094 ceph-mon[29291]: from='client.57002 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi094-78099-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:20.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:20 smithi094 ceph-mon[29291]: osdmap e308: 8 total, 8 up, 8 in 2024-03-12T03:20:20.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi094-78099-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:20.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:20 smithi094 ceph-mon[29291]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi094-78099-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:20 smithi151 ceph-mon[32923]: pgmap v419: 292 pgs: 292 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-12T03:20:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:20 smithi151 ceph-mon[32923]: from='client.57002 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi094-78099-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:20 smithi151 ceph-mon[32923]: osdmap e308: 8 total, 8 up, 8 in 2024-03-12T03:20:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi094-78099-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:20 smithi151 ceph-mon[32923]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi094-78099-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:20.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:20:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:20:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:20:21.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:21 smithi094 ceph-mon[29291]: osdmap e309: 8 total, 8 up, 8 in 2024-03-12T03:20:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:21 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:21.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:21.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:21 smithi094 ceph-mon[40484]: osdmap e309: 8 total, 8 up, 8 in 2024-03-12T03:20:21.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:21 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:21.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:21.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:21 smithi151 ceph-mon[32923]: osdmap e309: 8 total, 8 up, 8 in 2024-03-12T03:20:21.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:21 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[29291]: pgmap v422: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[29291]: from='client.57002 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi094-78099-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi094-78099-58"}]': finished 2024-03-12T03:20:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[29291]: osdmap e310: 8 total, 8 up, 8 in 2024-03-12T03:20:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:22.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[40484]: pgmap v422: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[40484]: from='client.57002 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi094-78099-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi094-78099-58"}]': finished 2024-03-12T03:20:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[40484]: osdmap e310: 8 total, 8 up, 8 in 2024-03-12T03:20:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:22 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:22 smithi151 ceph-mon[32923]: pgmap v422: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:22 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:22 smithi151 ceph-mon[32923]: from='client.57002 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi094-78099-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi094-78099-58"}]': finished 2024-03-12T03:20:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:22 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:22.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:22 smithi151 ceph-mon[32923]: osdmap e310: 8 total, 8 up, 8 in 2024-03-12T03:20:22.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:22.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:22 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:23.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:23 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:23.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-12T03:20:23.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:23 smithi094 ceph-mon[29291]: osdmap e311: 8 total, 8 up, 8 in 2024-03-12T03:20:23.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:23 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-12T03:20:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:23 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-12T03:20:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:23 smithi094 ceph-mon[40484]: osdmap e311: 8 total, 8 up, 8 in 2024-03-12T03:20:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:23 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-12T03:20:23.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:23 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:23.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-12T03:20:23.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:23 smithi151 ceph-mon[32923]: osdmap e311: 8 total, 8 up, 8 in 2024-03-12T03:20:23.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:23 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-12T03:20:24.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[29291]: pgmap v425: 300 pgs: 36 unknown, 264 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:24.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_count","val": "3"}]': finished 2024-03-12T03:20:24.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[29291]: osdmap e312: 8 total, 8 up, 8 in 2024-03-12T03:20:24.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-12T03:20:24.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-12T03:20:24.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:24.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[29291]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:24.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[40484]: pgmap v425: 300 pgs: 36 unknown, 264 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:24.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_count","val": "3"}]': finished 2024-03-12T03:20:24.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[40484]: osdmap e312: 8 total, 8 up, 8 in 2024-03-12T03:20:24.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-12T03:20:24.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-12T03:20:24.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:24.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:24 smithi094 ceph-mon[40484]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:24 smithi151 ceph-mon[32923]: pgmap v425: 300 pgs: 36 unknown, 264 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:24 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_count","val": "3"}]': finished 2024-03-12T03:20:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:24 smithi151 ceph-mon[32923]: osdmap e312: 8 total, 8 up, 8 in 2024-03-12T03:20:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:24 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-12T03:20:24.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:24 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-12T03:20:24.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:24 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:24.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:24 smithi151 ceph-mon[32923]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:25 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_period","val": "3"}]': finished 2024-03-12T03:20:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:25 smithi151 ceph-mon[32923]: from='client.57002 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]': finished 2024-03-12T03:20:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:20:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:25 smithi151 ceph-mon[32923]: osdmap e313: 8 total, 8 up, 8 in 2024-03-12T03:20:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:25 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:20:25.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:25 smithi151 ceph-mon[32923]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:25.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_period","val": "3"}]': finished 2024-03-12T03:20:25.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[29291]: from='client.57002 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]': finished 2024-03-12T03:20:25.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:20:25.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[29291]: osdmap e313: 8 total, 8 up, 8 in 2024-03-12T03:20:25.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:25.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:20:25.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[29291]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:25.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_period","val": "3"}]': finished 2024-03-12T03:20:25.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[40484]: from='client.57002 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi094-78099-58"}]': finished 2024-03-12T03:20:25.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:20:25.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[40484]: osdmap e313: 8 total, 8 up, 8 in 2024-03-12T03:20:25.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2397964248' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:25.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:20:25.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:25 smithi094 ceph-mon[40484]: from='client.57002 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]: dispatch 2024-03-12T03:20:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[40484]: pgmap v428: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:20:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[40484]: from='client.57002 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]': finished 2024-03-12T03:20:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[40484]: osdmap e314: 8 total, 8 up, 8 in 2024-03-12T03:20:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-12T03:20:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-12T03:20:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[40484]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[40484]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:26.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:26.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[40484]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[29291]: pgmap v428: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:20:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[29291]: from='client.57002 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]': finished 2024-03-12T03:20:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[29291]: osdmap e314: 8 total, 8 up, 8 in 2024-03-12T03:20:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-12T03:20:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-12T03:20:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[29291]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:26.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:26.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[29291]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:26.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:26.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:26 smithi094 ceph-mon[29291]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:26 smithi151 ceph-mon[32923]: pgmap v428: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:26 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:20:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:26 smithi151 ceph-mon[32923]: from='client.57002 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi094-78099-58"}]': finished 2024-03-12T03:20:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:26 smithi151 ceph-mon[32923]: osdmap e314: 8 total, 8 up, 8 in 2024-03-12T03:20:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-12T03:20:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:26 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-12T03:20:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:26 smithi151 ceph-mon[32923]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:26 smithi151 ceph-mon[32923]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:26.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:26 smithi151 ceph-mon[32923]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:27.890 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:27 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-12T03:20:27.890 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:27 smithi151 ceph-mon[32923]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:27.890 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:27 smithi151 ceph-mon[32923]: osdmap e315: 8 total, 8 up, 8 in 2024-03-12T03:20:27.890 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi094-78099-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:27.890 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:27 smithi151 ceph-mon[32923]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi094-78099-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:27.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:27 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-12T03:20:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:27 smithi094 ceph-mon[29291]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:27 smithi094 ceph-mon[29291]: osdmap e315: 8 total, 8 up, 8 in 2024-03-12T03:20:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi094-78099-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:27 smithi094 ceph-mon[29291]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi094-78099-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:27 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-12T03:20:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:27 smithi094 ceph-mon[40484]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:27 smithi094 ceph-mon[40484]: osdmap e315: 8 total, 8 up, 8 in 2024-03-12T03:20:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi094-78099-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:27 smithi094 ceph-mon[40484]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi094-78099-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:28.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:20:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:20:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:28 smithi094 ceph-mon[29291]: pgmap v431: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:28 smithi094 ceph-mon[29291]: osdmap e316: 8 total, 8 up, 8 in 2024-03-12T03:20:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:28 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:28 smithi094 ceph-mon[29291]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi094-78099-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]': finished 2024-03-12T03:20:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:28 smithi094 ceph-mon[29291]: osdmap e317: 8 total, 8 up, 8 in 2024-03-12T03:20:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:28 smithi094 ceph-mon[40484]: pgmap v431: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:28 smithi094 ceph-mon[40484]: osdmap e316: 8 total, 8 up, 8 in 2024-03-12T03:20:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:28 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:28 smithi094 ceph-mon[40484]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi094-78099-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]': finished 2024-03-12T03:20:28.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:28 smithi094 ceph-mon[40484]: osdmap e317: 8 total, 8 up, 8 in 2024-03-12T03:20:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:28 smithi151 ceph-mon[32923]: pgmap v431: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:28 smithi151 ceph-mon[32923]: osdmap e316: 8 total, 8 up, 8 in 2024-03-12T03:20:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:28 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:28 smithi151 ceph-mon[32923]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi094-78099-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]': finished 2024-03-12T03:20:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:28 smithi151 ceph-mon[32923]: osdmap e317: 8 total, 8 up, 8 in 2024-03-12T03:20:29.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:29 smithi094 ceph-mon[29291]: osdmap e318: 8 total, 8 up, 8 in 2024-03-12T03:20:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:29 smithi094 ceph-mon[40484]: osdmap e318: 8 total, 8 up, 8 in 2024-03-12T03:20:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:29 smithi151 ceph-mon[32923]: osdmap e318: 8 total, 8 up, 8 in 2024-03-12T03:20:30.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:20:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:20:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:20:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:30 smithi094 ceph-mon[29291]: pgmap v434: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:20:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:30 smithi094 ceph-mon[29291]: osdmap e319: 8 total, 8 up, 8 in 2024-03-12T03:20:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:30 smithi094 ceph-mon[29291]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:30 smithi094 ceph-mon[40484]: pgmap v434: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:20:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:30 smithi094 ceph-mon[40484]: osdmap e319: 8 total, 8 up, 8 in 2024-03-12T03:20:30.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:30.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:30 smithi094 ceph-mon[40484]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:30 smithi151 ceph-mon[32923]: pgmap v434: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:20:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:30 smithi151 ceph-mon[32923]: osdmap e319: 8 total, 8 up, 8 in 2024-03-12T03:20:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:30 smithi151 ceph-mon[32923]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:32.300 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:7462ddf6:::1o.RoundTripAppendPP (2999 ms) 2024-03-12T03:20:32.300 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-12T03:20:32.301 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3005 ms) 2024-03-12T03:20:32.301 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-12T03:20:32.301 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3246 ms) 2024-03-12T03:20:32.301 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-12T03:20:32.301 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2755 ms) 2024-03-12T03:20:32.301 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-12T03:20:32.301 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: setting pool EIO 2024-03-12T03:20:32.301 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: max_success 100, min_failed 101 2024-03-12T03:20:32.301 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4241 ms) 2024-03-12T03:20:32.301 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-12T03:20:32.301 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (2776 ms) 2024-03-12T03:20:32.301 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (128493 ms total) 2024-03-12T03:20:32.301 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3014 ms) 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9033 ms) 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14945 ms) 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3007 ms) 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29999 ms total) 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-12T03:20:32.302 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3008 ms) 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3008 ms total) 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13989 ms) 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7023 ms) 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6984 ms) 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7018 ms) 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3009 ms) 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-12T03:20:32.303 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7016 ms) 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7025 ms) 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6935 ms) 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7011 ms) 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7014 ms) 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7044 ms) 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7045 ms) 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-12T03:20:32.304 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7012 ms) 2024-03-12T03:20:32.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:32 smithi094 ceph-mon[29291]: pgmap v437: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:20:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:32 smithi094 ceph-mon[29291]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]': finished 2024-03-12T03:20:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:32 smithi094 ceph-mon[29291]: osdmap e320: 8 total, 8 up, 8 in 2024-03-12T03:20:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:32 smithi094 ceph-mon[29291]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:32 smithi094 ceph-mon[40484]: pgmap v437: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:20:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:32 smithi094 ceph-mon[40484]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]': finished 2024-03-12T03:20:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:32 smithi094 ceph-mon[40484]: osdmap e320: 8 total, 8 up, 8 in 2024-03-12T03:20:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:32 smithi094 ceph-mon[40484]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:32 smithi151 ceph-mon[32923]: pgmap v437: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:20:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:32 smithi151 ceph-mon[32923]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi094-78099-59"}]': finished 2024-03-12T03:20:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:32 smithi151 ceph-mon[32923]: osdmap e320: 8 total, 8 up, 8 in 2024-03-12T03:20:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/584493373' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:32 smithi151 ceph-mon[32923]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]: dispatch 2024-03-12T03:20:33.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:33 smithi094 ceph-mon[29291]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]': finished 2024-03-12T03:20:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:33 smithi094 ceph-mon[29291]: osdmap e321: 8 total, 8 up, 8 in 2024-03-12T03:20:33.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:33 smithi094 ceph-mon[40484]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]': finished 2024-03-12T03:20:33.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:33 smithi094 ceph-mon[40484]: osdmap e321: 8 total, 8 up, 8 in 2024-03-12T03:20:33.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:33 smithi151 ceph-mon[32923]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi094-78099-59"}]': finished 2024-03-12T03:20:33.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:33 smithi151 ceph-mon[32923]: osdmap e321: 8 total, 8 up, 8 in 2024-03-12T03:20:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:34 smithi094 ceph-mon[40484]: pgmap v440: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:34 smithi094 ceph-mon[40484]: osdmap e322: 8 total, 8 up, 8 in 2024-03-12T03:20:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1284638799' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:34 smithi094 ceph-mon[40484]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:34 smithi094 ceph-mon[29291]: pgmap v440: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:34 smithi094 ceph-mon[29291]: osdmap e322: 8 total, 8 up, 8 in 2024-03-12T03:20:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1284638799' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:34 smithi094 ceph-mon[29291]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:34.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:34 smithi151 ceph-mon[32923]: pgmap v440: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:34 smithi151 ceph-mon[32923]: osdmap e322: 8 total, 8 up, 8 in 2024-03-12T03:20:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1284638799' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:34 smithi151 ceph-mon[32923]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:34.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:35 smithi094 ceph-mon[40484]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:35 smithi094 ceph-mon[40484]: osdmap e323: 8 total, 8 up, 8 in 2024-03-12T03:20:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:35 smithi094 ceph-mon[29291]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:35 smithi094 ceph-mon[29291]: osdmap e323: 8 total, 8 up, 8 in 2024-03-12T03:20:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:35 smithi151 ceph-mon[32923]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi094-78099-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:35 smithi151 ceph-mon[32923]: osdmap e323: 8 total, 8 up, 8 in 2024-03-12T03:20:36.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:36 smithi094 ceph-mon[29291]: pgmap v443: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-12T03:20:36.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:36 smithi094 ceph-mon[29291]: osdmap e324: 8 total, 8 up, 8 in 2024-03-12T03:20:36.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:36.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:36.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi094-78099-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:36 smithi094 ceph-mon[40484]: pgmap v443: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-12T03:20:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:36 smithi094 ceph-mon[40484]: osdmap e324: 8 total, 8 up, 8 in 2024-03-12T03:20:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:36.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:36.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi094-78099-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:36 smithi151 ceph-mon[32923]: pgmap v443: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-12T03:20:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:36 smithi151 ceph-mon[32923]: osdmap e324: 8 total, 8 up, 8 in 2024-03-12T03:20:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:36.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi094-78099-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi094-78099-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:37 smithi094 ceph-mon[29291]: osdmap e325: 8 total, 8 up, 8 in 2024-03-12T03:20:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi094-78099-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:37.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi094-78099-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:37.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:37 smithi094 ceph-mon[40484]: osdmap e325: 8 total, 8 up, 8 in 2024-03-12T03:20:37.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi094-78099-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi094-78099-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:37 smithi151 ceph-mon[32923]: osdmap e325: 8 total, 8 up, 8 in 2024-03-12T03:20:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi094-78099-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:38.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:20:37 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:20:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:38 smithi094 ceph-mon[40484]: pgmap v446: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-12T03:20:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:38 smithi094 ceph-mon[40484]: osdmap e326: 8 total, 8 up, 8 in 2024-03-12T03:20:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:38 smithi094 ceph-mon[29291]: pgmap v446: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-12T03:20:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:38 smithi094 ceph-mon[29291]: osdmap e326: 8 total, 8 up, 8 in 2024-03-12T03:20:38.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:38 smithi151 ceph-mon[32923]: pgmap v446: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-12T03:20:38.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:38 smithi151 ceph-mon[32923]: osdmap e326: 8 total, 8 up, 8 in 2024-03-12T03:20:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi094-78099-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi094-78099-61"}]': finished 2024-03-12T03:20:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[29291]: osdmap e327: 8 total, 8 up, 8 in 2024-03-12T03:20:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45"}]: dispatch 2024-03-12T03:20:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45"}]: dispatch 2024-03-12T03:20:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi094-78099-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi094-78099-61"}]': finished 2024-03-12T03:20:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[40484]: osdmap e327: 8 total, 8 up, 8 in 2024-03-12T03:20:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45"}]: dispatch 2024-03-12T03:20:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:39 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45"}]: dispatch 2024-03-12T03:20:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi094-78099-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi094-78099-61"}]': finished 2024-03-12T03:20:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:39 smithi151 ceph-mon[32923]: osdmap e327: 8 total, 8 up, 8 in 2024-03-12T03:20:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:39.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:39 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:39.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45"}]: dispatch 2024-03-12T03:20:39.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:39 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45"}]: dispatch 2024-03-12T03:20:40.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:20:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:20:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:20:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:40 smithi094 ceph-mon[29291]: pgmap v449: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-12T03:20:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:40 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:40 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45"}]': finished 2024-03-12T03:20:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:40 smithi094 ceph-mon[29291]: osdmap e328: 8 total, 8 up, 8 in 2024-03-12T03:20:40.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:40 smithi094 ceph-mon[40484]: pgmap v449: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-12T03:20:40.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:40 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:40.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:40 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45"}]': finished 2024-03-12T03:20:40.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:40 smithi094 ceph-mon[40484]: osdmap e328: 8 total, 8 up, 8 in 2024-03-12T03:20:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:40 smithi151 ceph-mon[32923]: pgmap v449: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-12T03:20:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:40 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:40 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-45"}]': finished 2024-03-12T03:20:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:40 smithi151 ceph-mon[32923]: osdmap e328: 8 total, 8 up, 8 in 2024-03-12T03:20:41.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:41 smithi094 ceph-mon[29291]: osdmap e329: 8 total, 8 up, 8 in 2024-03-12T03:20:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:41 smithi094 ceph-mon[40484]: osdmap e329: 8 total, 8 up, 8 in 2024-03-12T03:20:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:41 smithi151 ceph-mon[32923]: osdmap e329: 8 total, 8 up, 8 in 2024-03-12T03:20:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:42 smithi094 ceph-mon[29291]: pgmap v452: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi094-78099-61"}]': finished 2024-03-12T03:20:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:42 smithi094 ceph-mon[29291]: osdmap e330: 8 total, 8 up, 8 in 2024-03-12T03:20:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:42 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:42.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:42 smithi094 ceph-mon[40484]: pgmap v452: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:42.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi094-78099-61"}]': finished 2024-03-12T03:20:42.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:42 smithi094 ceph-mon[40484]: osdmap e330: 8 total, 8 up, 8 in 2024-03-12T03:20:42.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:42.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:42.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:42 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:42.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:42 smithi151 ceph-mon[32923]: pgmap v452: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:42.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi094-78099-61"}]': finished 2024-03-12T03:20:42.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:42 smithi151 ceph-mon[32923]: osdmap e330: 8 total, 8 up, 8 in 2024-03-12T03:20:42.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:42.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi094-78099-61"}]: dispatch 2024-03-12T03:20:42.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:42 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi094-78099-61"}]': finished 2024-03-12T03:20:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[29291]: osdmap e331: 8 total, 8 up, 8 in 2024-03-12T03:20:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:43.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:43.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi094-78099-61"}]': finished 2024-03-12T03:20:43.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:43.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[40484]: osdmap e331: 8 total, 8 up, 8 in 2024-03-12T03:20:43.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:43.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:43.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:43.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:43.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3482833871' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi094-78099-61"}]': finished 2024-03-12T03:20:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:43 smithi151 ceph-mon[32923]: osdmap e331: 8 total, 8 up, 8 in 2024-03-12T03:20:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:43.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:43.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:43.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:43.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[40484]: pgmap v455: 292 pgs: 27 unknown, 265 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[40484]: osdmap e332: 8 total, 8 up, 8 in 2024-03-12T03:20:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi094-78099-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[29291]: pgmap v455: 292 pgs: 27 unknown, 265 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:44.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:44.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:44.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[29291]: osdmap e332: 8 total, 8 up, 8 in 2024-03-12T03:20:44.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:44.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi094-78099-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:44.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:44 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:44 smithi151 ceph-mon[32923]: pgmap v455: 292 pgs: 27 unknown, 265 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:44 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:44 smithi151 ceph-mon[32923]: osdmap e332: 8 total, 8 up, 8 in 2024-03-12T03:20:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:44.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi094-78099-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:44.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:44 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-47"}]': finished 2024-03-12T03:20:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-47", "mode": "writeback"}]: dispatch 2024-03-12T03:20:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[40484]: osdmap e333: 8 total, 8 up, 8 in 2024-03-12T03:20:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-47", "mode": "writeback"}]: dispatch 2024-03-12T03:20:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[40484]: pgmap v458: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-12T03:20:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:20:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi094-78099-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi094-78099-62"}]': finished 2024-03-12T03:20:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-47", "mode": "writeback"}]': finished 2024-03-12T03:20:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[40484]: osdmap e334: 8 total, 8 up, 8 in 2024-03-12T03:20:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:20:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:20:45.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-47"}]': finished 2024-03-12T03:20:45.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-47", "mode": "writeback"}]: dispatch 2024-03-12T03:20:45.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[29291]: osdmap e333: 8 total, 8 up, 8 in 2024-03-12T03:20:45.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-47", "mode": "writeback"}]: dispatch 2024-03-12T03:20:45.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[29291]: pgmap v458: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-12T03:20:45.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:20:45.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi094-78099-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi094-78099-62"}]': finished 2024-03-12T03:20:45.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-47", "mode": "writeback"}]': finished 2024-03-12T03:20:45.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[29291]: osdmap e334: 8 total, 8 up, 8 in 2024-03-12T03:20:45.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:20:45.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:45 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:20:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:45 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-47"}]': finished 2024-03-12T03:20:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-47", "mode": "writeback"}]: dispatch 2024-03-12T03:20:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:45 smithi151 ceph-mon[32923]: osdmap e333: 8 total, 8 up, 8 in 2024-03-12T03:20:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:45 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-47", "mode": "writeback"}]: dispatch 2024-03-12T03:20:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:45 smithi151 ceph-mon[32923]: pgmap v458: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-12T03:20:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:45 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:20:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi094-78099-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi094-78099-62"}]': finished 2024-03-12T03:20:45.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:45 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-47", "mode": "writeback"}]': finished 2024-03-12T03:20:45.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:45 smithi151 ceph-mon[32923]: osdmap e334: 8 total, 8 up, 8 in 2024-03-12T03:20:45.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:20:45.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:45 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:20:47.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:47 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:20:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:47 smithi094 ceph-mon[29291]: osdmap e335: 8 total, 8 up, 8 in 2024-03-12T03:20:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:47 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:47 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:20:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:47 smithi094 ceph-mon[40484]: osdmap e335: 8 total, 8 up, 8 in 2024-03-12T03:20:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:47 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:47 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:20:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:47 smithi151 ceph-mon[32923]: osdmap e335: 8 total, 8 up, 8 in 2024-03-12T03:20:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:47 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:20:48.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:20:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:20:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[40484]: pgmap v461: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-12T03:20:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:20:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[40484]: osdmap e336: 8 total, 8 up, 8 in 2024-03-12T03:20:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:20:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:20:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[29291]: pgmap v461: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-12T03:20:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:20:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[29291]: osdmap e336: 8 total, 8 up, 8 in 2024-03-12T03:20:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:20:48.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:48 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:20:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:48 smithi151 ceph-mon[32923]: pgmap v461: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-12T03:20:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:48 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:48 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:20:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:48 smithi151 ceph-mon[32923]: osdmap e336: 8 total, 8 up, 8 in 2024-03-12T03:20:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:20:48.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:48 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:20:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:20:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62"}]': finished 2024-03-12T03:20:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:20:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[40484]: osdmap e337: 8 total, 8 up, 8 in 2024-03-12T03:20:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:20:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:20:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:49.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi094-78099-62"}]': finished 2024-03-12T03:20:49.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-12T03:20:49.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[40484]: osdmap e338: 8 total, 8 up, 8 in 2024-03-12T03:20:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:20:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62"}]': finished 2024-03-12T03:20:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:20:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[29291]: osdmap e337: 8 total, 8 up, 8 in 2024-03-12T03:20:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:20:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:20:49.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:49.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi094-78099-62"}]': finished 2024-03-12T03:20:49.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-12T03:20:49.712 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:49 smithi094 ceph-mon[29291]: osdmap e338: 8 total, 8 up, 8 in 2024-03-12T03:20:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:49 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:20:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi094-78099-62"}]': finished 2024-03-12T03:20:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:20:49.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:49 smithi151 ceph-mon[32923]: osdmap e337: 8 total, 8 up, 8 in 2024-03-12T03:20:49.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi094-78099-62"}]: dispatch 2024-03-12T03:20:49.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:20:49.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:20:49.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:49 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:49.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1628650947' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi094-78099-62"}]': finished 2024-03-12T03:20:49.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-12T03:20:49.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:49 smithi151 ceph-mon[32923]: osdmap e338: 8 total, 8 up, 8 in 2024-03-12T03:20:50.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:20:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:20:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[40484]: pgmap v464: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-12T03:20:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-12T03:20:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[40484]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[40484]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi094-78099-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[40484]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi094-78099-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[29291]: pgmap v464: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-12T03:20:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-12T03:20:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[29291]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[29291]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi094-78099-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:50 smithi094 ceph-mon[29291]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi094-78099-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:50 smithi151 ceph-mon[32923]: pgmap v464: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-12T03:20:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:50 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-12T03:20:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:50 smithi151 ceph-mon[32923]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:50 smithi151 ceph-mon[32923]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi094-78099-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:50 smithi151 ceph-mon[32923]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi094-78099-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-12T03:20:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[29291]: from='client.57032 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi094-78099-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[29291]: osdmap e339: 8 total, 8 up, 8 in 2024-03-12T03:20:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-12T03:20:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi094-78099-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-12T03:20:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[29291]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi094-78099-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-12T03:20:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[40484]: from='client.57032 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi094-78099-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[40484]: osdmap e339: 8 total, 8 up, 8 in 2024-03-12T03:20:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-12T03:20:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi094-78099-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-12T03:20:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:51 smithi094 ceph-mon[40484]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi094-78099-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:51 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-12T03:20:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:51 smithi151 ceph-mon[32923]: from='client.57032 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi094-78099-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:51 smithi151 ceph-mon[32923]: osdmap e339: 8 total, 8 up, 8 in 2024-03-12T03:20:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-12T03:20:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi094-78099-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:51 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-12T03:20:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:51 smithi151 ceph-mon[32923]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi094-78099-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:52 smithi094 ceph-mon[29291]: pgmap v467: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-12T03:20:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:52 smithi094 ceph-mon[29291]: osdmap e340: 8 total, 8 up, 8 in 2024-03-12T03:20:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:52 smithi094 ceph-mon[40484]: pgmap v467: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-12T03:20:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:52 smithi094 ceph-mon[40484]: osdmap e340: 8 total, 8 up, 8 in 2024-03-12T03:20:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:52 smithi151 ceph-mon[32923]: pgmap v467: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:20:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-12T03:20:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:52 smithi151 ceph-mon[32923]: osdmap e340: 8 total, 8 up, 8 in 2024-03-12T03:20:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:53.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:53 smithi094 ceph-mon[29291]: from='client.57032 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi094-78099-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi094-78099-63"}]': finished 2024-03-12T03:20:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:53 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:20:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:53 smithi094 ceph-mon[29291]: osdmap e341: 8 total, 8 up, 8 in 2024-03-12T03:20:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:53 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:53 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:53 smithi094 ceph-mon[40484]: from='client.57032 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi094-78099-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi094-78099-63"}]': finished 2024-03-12T03:20:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:53 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:20:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:53 smithi094 ceph-mon[40484]: osdmap e341: 8 total, 8 up, 8 in 2024-03-12T03:20:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:53 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:53 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:53 smithi151 ceph-mon[32923]: from='client.57032 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi094-78099-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi094-78099-63"}]': finished 2024-03-12T03:20:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:53 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:20:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:53 smithi151 ceph-mon[32923]: osdmap e341: 8 total, 8 up, 8 in 2024-03-12T03:20:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:53 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:53 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:20:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[29291]: pgmap v470: 300 pgs: 1 creating+peering, 7 unknown, 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]': finished 2024-03-12T03:20:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[29291]: osdmap e342: 8 total, 8 up, 8 in 2024-03-12T03:20:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[40484]: pgmap v470: 300 pgs: 1 creating+peering, 7 unknown, 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]': finished 2024-03-12T03:20:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[40484]: osdmap e342: 8 total, 8 up, 8 in 2024-03-12T03:20:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:54 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:54 smithi151 ceph-mon[32923]: pgmap v470: 300 pgs: 1 creating+peering, 7 unknown, 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:20:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:54 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]': finished 2024-03-12T03:20:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:54 smithi151 ceph-mon[32923]: osdmap e342: 8 total, 8 up, 8 in 2024-03-12T03:20:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:54.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:54 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:20:54.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:54.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:54 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-47"}]: dispatch 2024-03-12T03:20:55.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:55 smithi094 ceph-mon[29291]: osdmap e343: 8 total, 8 up, 8 in 2024-03-12T03:20:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:55 smithi094 ceph-mon[29291]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:55 smithi094 ceph-mon[40484]: osdmap e343: 8 total, 8 up, 8 in 2024-03-12T03:20:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:55 smithi094 ceph-mon[40484]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:55 smithi151 ceph-mon[32923]: osdmap e343: 8 total, 8 up, 8 in 2024-03-12T03:20:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:55 smithi151 ceph-mon[32923]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[40484]: pgmap v473: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:20:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[40484]: from='client.57032 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]': finished 2024-03-12T03:20:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[40484]: osdmap e344: 8 total, 8 up, 8 in 2024-03-12T03:20:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[40484]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[29291]: pgmap v473: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:20:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[29291]: from='client.57032 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]': finished 2024-03-12T03:20:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[29291]: osdmap e344: 8 total, 8 up, 8 in 2024-03-12T03:20:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[29291]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:56 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:56 smithi151 ceph-mon[32923]: pgmap v473: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:20:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:56 smithi151 ceph-mon[32923]: from='client.57032 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi094-78099-63"}]': finished 2024-03-12T03:20:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:56 smithi151 ceph-mon[32923]: osdmap e344: 8 total, 8 up, 8 in 2024-03-12T03:20:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2706734605' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:56.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:56 smithi151 ceph-mon[32923]: from='client.57032 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]: dispatch 2024-03-12T03:20:56.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:56.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:56 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[29291]: from='client.57032 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]': finished 2024-03-12T03:20:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[29291]: osdmap e345: 8 total, 8 up, 8 in 2024-03-12T03:20:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[29291]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:57.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:57.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[29291]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:57.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi094-78099-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:57.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[29291]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi094-78099-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[40484]: from='client.57032 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]': finished 2024-03-12T03:20:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[40484]: osdmap e345: 8 total, 8 up, 8 in 2024-03-12T03:20:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:57.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[40484]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:57.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:57.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[40484]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:57.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi094-78099-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:57.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:57 smithi094 ceph-mon[40484]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi094-78099-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:57 smithi151 ceph-mon[32923]: from='client.57032 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi094-78099-63"}]': finished 2024-03-12T03:20:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:57 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:57 smithi151 ceph-mon[32923]: osdmap e345: 8 total, 8 up, 8 in 2024-03-12T03:20:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:57 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:20:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:57 smithi151 ceph-mon[32923]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:57 smithi151 ceph-mon[32923]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi094-78099-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:57.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:57 smithi151 ceph-mon[32923]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi094-78099-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:20:58.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:20:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:20:58.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[29291]: pgmap v476: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:20:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[29291]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi094-78099-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[29291]: osdmap e346: 8 total, 8 up, 8 in 2024-03-12T03:20:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi094-78099-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[29291]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi094-78099-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:20:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:20:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[40484]: pgmap v476: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:20:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:58.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[40484]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi094-78099-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:58.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[40484]: osdmap e346: 8 total, 8 up, 8 in 2024-03-12T03:20:58.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi094-78099-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:58.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[40484]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi094-78099-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:58.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:20:58.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:58 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:20:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:58 smithi151 ceph-mon[32923]: pgmap v476: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:20:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:58 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:20:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:58 smithi151 ceph-mon[32923]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi094-78099-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:20:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:58 smithi151 ceph-mon[32923]: osdmap e346: 8 total, 8 up, 8 in 2024-03-12T03:20:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi094-78099-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:58 smithi151 ceph-mon[32923]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi094-78099-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:20:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:20:58.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:58 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:20:59.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:59.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-49"}]': finished 2024-03-12T03:20:59.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-49", "mode": "readproxy"}]: dispatch 2024-03-12T03:20:59.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[29291]: pgmap v479: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:20:59.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[29291]: osdmap e347: 8 total, 8 up, 8 in 2024-03-12T03:20:59.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-49", "mode": "readproxy"}]: dispatch 2024-03-12T03:20:59.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:20:59.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[29291]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi094-78099-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi094-78099-64"}]': finished 2024-03-12T03:20:59.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-49", "mode": "readproxy"}]': finished 2024-03-12T03:20:59.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[29291]: osdmap e348: 8 total, 8 up, 8 in 2024-03-12T03:20:59.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:59.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-49"}]': finished 2024-03-12T03:20:59.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-49", "mode": "readproxy"}]: dispatch 2024-03-12T03:20:59.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[40484]: pgmap v479: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:20:59.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[40484]: osdmap e347: 8 total, 8 up, 8 in 2024-03-12T03:20:59.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-49", "mode": "readproxy"}]: dispatch 2024-03-12T03:20:59.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:20:59.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[40484]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi094-78099-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi094-78099-64"}]': finished 2024-03-12T03:20:59.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-49", "mode": "readproxy"}]': finished 2024-03-12T03:20:59.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:20:59 smithi094 ceph-mon[40484]: osdmap e348: 8 total, 8 up, 8 in 2024-03-12T03:20:59.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:20:59.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:59 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-49"}]': finished 2024-03-12T03:20:59.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-49", "mode": "readproxy"}]: dispatch 2024-03-12T03:20:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:59 smithi151 ceph-mon[32923]: pgmap v479: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:20:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:59 smithi151 ceph-mon[32923]: osdmap e347: 8 total, 8 up, 8 in 2024-03-12T03:20:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:59 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-49", "mode": "readproxy"}]: dispatch 2024-03-12T03:20:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:59 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:20:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:59 smithi151 ceph-mon[32923]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi094-78099-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi094-78099-64"}]': finished 2024-03-12T03:20:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:59 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-49", "mode": "readproxy"}]': finished 2024-03-12T03:20:59.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:20:59 smithi151 ceph-mon[32923]: osdmap e348: 8 total, 8 up, 8 in 2024-03-12T03:21:00.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:21:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:21:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:21:01.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:01 smithi094 ceph-mon[29291]: osdmap e349: 8 total, 8 up, 8 in 2024-03-12T03:21:01.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:01 smithi094 ceph-mon[40484]: osdmap e349: 8 total, 8 up, 8 in 2024-03-12T03:21:01.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:01 smithi151 ceph-mon[32923]: osdmap e349: 8 total, 8 up, 8 in 2024-03-12T03:21:02.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:02 smithi094 ceph-mon[29291]: pgmap v482: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:21:02.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:02 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:02.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:02 smithi094 ceph-mon[29291]: osdmap e350: 8 total, 8 up, 8 in 2024-03-12T03:21:02.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:21:02.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:02 smithi094 ceph-mon[29291]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:21:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:02 smithi094 ceph-mon[40484]: pgmap v482: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:21:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:02 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:02 smithi094 ceph-mon[40484]: osdmap e350: 8 total, 8 up, 8 in 2024-03-12T03:21:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:21:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:02 smithi094 ceph-mon[40484]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:21:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:02 smithi151 ceph-mon[32923]: pgmap v482: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:21:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:02 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:02 smithi151 ceph-mon[32923]: osdmap e350: 8 total, 8 up, 8 in 2024-03-12T03:21:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:21:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:02 smithi151 ceph-mon[32923]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:21:03.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:03 smithi094 ceph-mon[29291]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]': finished 2024-03-12T03:21:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:03 smithi094 ceph-mon[29291]: osdmap e351: 8 total, 8 up, 8 in 2024-03-12T03:21:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:21:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:03 smithi094 ceph-mon[29291]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:21:03.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:03 smithi094 ceph-mon[40484]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]': finished 2024-03-12T03:21:03.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:03 smithi094 ceph-mon[40484]: osdmap e351: 8 total, 8 up, 8 in 2024-03-12T03:21:03.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:21:03.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:03 smithi094 ceph-mon[40484]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:21:03.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:03 smithi151 ceph-mon[32923]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi094-78099-64"}]': finished 2024-03-12T03:21:03.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:03 smithi151 ceph-mon[32923]: osdmap e351: 8 total, 8 up, 8 in 2024-03-12T03:21:03.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/662142189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:21:03.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:03 smithi151 ceph-mon[32923]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]: dispatch 2024-03-12T03:21:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[29291]: pgmap v485: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[29291]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]': finished 2024-03-12T03:21:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[29291]: osdmap e352: 8 total, 8 up, 8 in 2024-03-12T03:21:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[29291]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[29291]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi094-78099-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:21:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[29291]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi094-78099-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:21:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:21:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[40484]: pgmap v485: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[40484]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]': finished 2024-03-12T03:21:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[40484]: osdmap e352: 8 total, 8 up, 8 in 2024-03-12T03:21:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[40484]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[40484]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi094-78099-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:21:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[40484]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi094-78099-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:21:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:21:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:04.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:04 smithi151 ceph-mon[32923]: pgmap v485: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:04 smithi151 ceph-mon[32923]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi094-78099-64"}]': finished 2024-03-12T03:21:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:04 smithi151 ceph-mon[32923]: osdmap e352: 8 total, 8 up, 8 in 2024-03-12T03:21:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:04 smithi151 ceph-mon[32923]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:04 smithi151 ceph-mon[32923]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi094-78099-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:21:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:04 smithi151 ceph-mon[32923]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi094-78099-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:21:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:04 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:21:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:05.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[29291]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi094-78099-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:21:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[29291]: osdmap e353: 8 total, 8 up, 8 in 2024-03-12T03:21:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi094-78099-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[29291]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi094-78099-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:21:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[40484]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi094-78099-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:21:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[40484]: osdmap e353: 8 total, 8 up, 8 in 2024-03-12T03:21:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi094-78099-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[40484]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi094-78099-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:05.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:05.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:05.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:05 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:21:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:05 smithi151 ceph-mon[32923]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi094-78099-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:21:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:05 smithi151 ceph-mon[32923]: osdmap e353: 8 total, 8 up, 8 in 2024-03-12T03:21:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi094-78099-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:05 smithi151 ceph-mon[32923]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi094-78099-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:05.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:05 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:21:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:06 smithi094 ceph-mon[29291]: pgmap v488: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:21:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:06 smithi094 ceph-mon[29291]: osdmap e354: 8 total, 8 up, 8 in 2024-03-12T03:21:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:06 smithi094 ceph-mon[40484]: pgmap v488: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:21:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:06 smithi094 ceph-mon[40484]: osdmap e354: 8 total, 8 up, 8 in 2024-03-12T03:21:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:06 smithi151 ceph-mon[32923]: pgmap v488: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:21:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:06 smithi151 ceph-mon[32923]: osdmap e354: 8 total, 8 up, 8 in 2024-03-12T03:21:07.934 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:07 smithi151 ceph-mon[32923]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi094-78099-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi094-78099-65"}]': finished 2024-03-12T03:21:07.934 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:07 smithi151 ceph-mon[32923]: osdmap e355: 8 total, 8 up, 8 in 2024-03-12T03:21:07.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:07 smithi094 ceph-mon[40484]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi094-78099-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi094-78099-65"}]': finished 2024-03-12T03:21:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:07 smithi094 ceph-mon[40484]: osdmap e355: 8 total, 8 up, 8 in 2024-03-12T03:21:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:07 smithi094 ceph-mon[29291]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi094-78099-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi094-78099-65"}]': finished 2024-03-12T03:21:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:07 smithi094 ceph-mon[29291]: osdmap e355: 8 total, 8 up, 8 in 2024-03-12T03:21:08.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:21:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:21:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:08 smithi094 ceph-mon[29291]: pgmap v491: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:21:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:08 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:08 smithi094 ceph-mon[29291]: osdmap e356: 8 total, 8 up, 8 in 2024-03-12T03:21:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:08 smithi094 ceph-mon[40484]: pgmap v491: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:21:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:08 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:08 smithi094 ceph-mon[40484]: osdmap e356: 8 total, 8 up, 8 in 2024-03-12T03:21:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:08 smithi151 ceph-mon[32923]: pgmap v491: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:21:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:08 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:08 smithi151 ceph-mon[32923]: osdmap e356: 8 total, 8 up, 8 in 2024-03-12T03:21:09.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:09 smithi094 ceph-mon[29291]: osdmap e357: 8 total, 8 up, 8 in 2024-03-12T03:21:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:09 smithi094 ceph-mon[29291]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:09 smithi094 ceph-mon[40484]: osdmap e357: 8 total, 8 up, 8 in 2024-03-12T03:21:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:09 smithi094 ceph-mon[40484]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:09 smithi151 ceph-mon[32923]: osdmap e357: 8 total, 8 up, 8 in 2024-03-12T03:21:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:09 smithi151 ceph-mon[32923]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:10.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:21:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:21:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[29291]: pgmap v494: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:21:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[29291]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]': finished 2024-03-12T03:21:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[29291]: osdmap e358: 8 total, 8 up, 8 in 2024-03-12T03:21:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[29291]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:10.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[40484]: pgmap v494: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:21:10.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[40484]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]': finished 2024-03-12T03:21:10.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[40484]: osdmap e358: 8 total, 8 up, 8 in 2024-03-12T03:21:10.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:10.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[40484]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:10.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:10.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:10 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:10 smithi151 ceph-mon[32923]: pgmap v494: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:21:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:10 smithi151 ceph-mon[32923]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi094-78099-65"}]': finished 2024-03-12T03:21:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:10 smithi151 ceph-mon[32923]: osdmap e358: 8 total, 8 up, 8 in 2024-03-12T03:21:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4222614154' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:10 smithi151 ceph-mon[32923]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]: dispatch 2024-03-12T03:21:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:10.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:10 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[29291]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]': finished 2024-03-12T03:21:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:21:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[29291]: osdmap e359: 8 total, 8 up, 8 in 2024-03-12T03:21:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:21:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:21:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi094-78099-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:21:11.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[40484]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]': finished 2024-03-12T03:21:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:21:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[40484]: osdmap e359: 8 total, 8 up, 8 in 2024-03-12T03:21:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:21:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:21:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi094-78099-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:21:11.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:11 smithi151 ceph-mon[32923]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi094-78099-65"}]': finished 2024-03-12T03:21:11.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:11 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:21:11.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:11 smithi151 ceph-mon[32923]: osdmap e359: 8 total, 8 up, 8 in 2024-03-12T03:21:11.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:21:11.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:11 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:21:11.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:11.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:11.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi094-78099-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:21:12.782 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP63:head 2024-03-12T03:21:12.782 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:5d165639:::164:head 2024-03-12T03:21:12.782 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:f43765fc:::165:head 2024-03-12T03:21:12.782 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:b4c720e9:::166:head 2024-03-12T03:21:12.782 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:e694b040:::167:head 2024-03-12T03:21:12.782 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:afa38db2:::168:head 2024-03-12T03:21:12.782 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:77ba9f53:::169:head 2024-03-12T03:21:12.782 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:87495034:::170:head 2024-03-12T03:21:12.782 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:7c96bf0e:::171:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:dbe346cc:::172:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:e943ec24:::173:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:f97a9c0c:::174:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:6f26e74d:::175:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:4f95e106:::176:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:0e6f2f8f:::177:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:05db05f1:::178:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:38a78d66:::179:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:d095610b:::180:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:a1a9d709:::181:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:1e5d39db:::182:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:f7df4fb9:::183:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:03a7f161:::184:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:ba70721e:::185:head 2024-03-12T03:21:12.783 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:28e5662d:::186:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:973d52de:::187:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:4303eb1c:::188:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:b990b48e:::189:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:29b8165b:::190:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:3547f197:::191:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:7e260936:::192:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:1abec7b1:::193:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:10fdda93:::194:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:15817eea:::195:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:770bab57:::196:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:ed9e13e7:::197:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:71471a8f:::198:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: checking for 262:10fb1d02:::199:head 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8047 ms) 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-12T03:21:12.784 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710213627,0 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: first is 1710213627 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710213627,0 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710213627,0 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710213627,1710213629,1710213630,0 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710213627,1710213629,1710213630,0 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710213627,1710213629,1710213630,0 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710213627,1710213629,1710213630,1710213632,1710213633,0 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710213627,1710213629,1710213630,1710213632,1710213633,0 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710213627,1710213629,1710213630,1710213632,1710213633,0 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710213627,1710213629,1710213630,1710213632,1710213633,1710213635,1710213636,0 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710213627,1710213629,1710213630,1710213632,1710213633,1710213635,1710213636,0 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710213627,1710213629,1710213630,1710213632,1710213633,1710213635,1710213636,0 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710213630,1710213632,1710213633,1710213635,1710213636,1710213638,1710213639,0 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: first now 1710213630, trimmed 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21107 ms) 2024-03-12T03:21:12.785 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-12T03:21:12.786 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: foo0 2024-03-12T03:21:12.786 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-12T03:21:12.786 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14084 ms) 2024-03-12T03:21:12.786 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-12T03:21:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[40484]: pgmap v497: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:21:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:21:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]': finished 2024-03-12T03:21:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi094-78099-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:21:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[40484]: osdmap e360: 8 total, 8 up, 8 in 2024-03-12T03:21:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi094-78099-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:21:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:21:12.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[29291]: pgmap v497: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:21:12.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:21:12.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]': finished 2024-03-12T03:21:12.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi094-78099-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:21:12.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[29291]: osdmap e360: 8 total, 8 up, 8 in 2024-03-12T03:21:12.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi094-78099-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:12.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:12.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:12.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:21:12.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:12 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:21:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:12 smithi151 ceph-mon[32923]: pgmap v497: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:21:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:12 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:21:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:12 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]': finished 2024-03-12T03:21:12.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi094-78099-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:21:12.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:12 smithi151 ceph-mon[32923]: osdmap e360: 8 total, 8 up, 8 in 2024-03-12T03:21:12.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi094-78099-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:12.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:12.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:12 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:12.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:21:12.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:12 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-49"}]: dispatch 2024-03-12T03:21:13.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:13 smithi094 ceph-mon[29291]: osdmap e361: 8 total, 8 up, 8 in 2024-03-12T03:21:13.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:13 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:13.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:13 smithi094 ceph-mon[40484]: osdmap e361: 8 total, 8 up, 8 in 2024-03-12T03:21:13.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:13 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:13.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:13 smithi151 ceph-mon[32923]: osdmap e361: 8 total, 8 up, 8 in 2024-03-12T03:21:13.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:13 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:14 smithi151 ceph-mon[32923]: pgmap v500: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi094-78099-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi094-78099-66"}]': finished 2024-03-12T03:21:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:14 smithi151 ceph-mon[32923]: osdmap e362: 8 total, 8 up, 8 in 2024-03-12T03:21:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:14 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:14 smithi094 ceph-mon[40484]: pgmap v500: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi094-78099-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi094-78099-66"}]': finished 2024-03-12T03:21:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:14 smithi094 ceph-mon[40484]: osdmap e362: 8 total, 8 up, 8 in 2024-03-12T03:21:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:14 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:14 smithi094 ceph-mon[29291]: pgmap v500: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi094-78099-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi094-78099-66"}]': finished 2024-03-12T03:21:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:14 smithi094 ceph-mon[29291]: osdmap e362: 8 total, 8 up, 8 in 2024-03-12T03:21:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:14.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:14 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:15.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:15 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:15 smithi094 ceph-mon[29291]: osdmap e363: 8 total, 8 up, 8 in 2024-03-12T03:21:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:15 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:21:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:15 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:21:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:15 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:15 smithi094 ceph-mon[40484]: osdmap e363: 8 total, 8 up, 8 in 2024-03-12T03:21:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:15 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:21:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:15 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:21:15.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:15 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:15.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:15 smithi151 ceph-mon[32923]: osdmap e363: 8 total, 8 up, 8 in 2024-03-12T03:21:15.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:15 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:21:15.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:15 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:21:16.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:16 smithi094 ceph-mon[29291]: pgmap v503: 300 pgs: 40 unknown, 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:21:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:16 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:21:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:16 smithi094 ceph-mon[29291]: osdmap e364: 8 total, 8 up, 8 in 2024-03-12T03:21:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:16 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:16 smithi094 ceph-mon[40484]: pgmap v503: 300 pgs: 40 unknown, 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:21:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:16 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:21:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:16 smithi094 ceph-mon[40484]: osdmap e364: 8 total, 8 up, 8 in 2024-03-12T03:21:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:16 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:16.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:16 smithi151 ceph-mon[32923]: pgmap v503: 300 pgs: 40 unknown, 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:21:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:16 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:21:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:16 smithi151 ceph-mon[32923]: osdmap e364: 8 total, 8 up, 8 in 2024-03-12T03:21:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:16 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:17.578 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ 2 2024-03-12T03:21:17.578 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3025 ms) 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7038 ms) 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7038 ms) 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7044 ms) 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7045 ms) 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7035 ms) 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7048 ms) 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139399 ms total) 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: 2024-03-12T03:21:17.579 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-12T03:21:17.580 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (300899 ms total) 2024-03-12T03:21:17.580 INFO:tasks.workunit.client.0.smithi094.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-12T03:21:17.861 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-51"}]': finished 2024-03-12T03:21:17.861 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi094-78099-66"}]': finished 2024-03-12T03:21:17.861 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[29291]: osdmap e365: 8 total, 8 up, 8 in 2024-03-12T03:21:17.861 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-51", "mode": "writeback"}]: dispatch 2024-03-12T03:21:17.861 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:17.861 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-51", "mode": "writeback"}]: dispatch 2024-03-12T03:21:17.861 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:21:17.862 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi094-78099-66"}]': finished 2024-03-12T03:21:17.862 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-51"}]': finished 2024-03-12T03:21:17.862 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi094-78099-66"}]': finished 2024-03-12T03:21:17.862 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[40484]: osdmap e365: 8 total, 8 up, 8 in 2024-03-12T03:21:17.862 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-51", "mode": "writeback"}]: dispatch 2024-03-12T03:21:17.862 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:17.862 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-51", "mode": "writeback"}]: dispatch 2024-03-12T03:21:17.862 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:21:17.862 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi094-78099-66"}]': finished 2024-03-12T03:21:17.945 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:17 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-51"}]': finished 2024-03-12T03:21:17.945 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi094-78099-66"}]': finished 2024-03-12T03:21:17.945 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:17 smithi151 ceph-mon[32923]: osdmap e365: 8 total, 8 up, 8 in 2024-03-12T03:21:17.945 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-51", "mode": "writeback"}]: dispatch 2024-03-12T03:21:17.945 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi094-78099-66"}]: dispatch 2024-03-12T03:21:17.945 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:17 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-51", "mode": "writeback"}]: dispatch 2024-03-12T03:21:17.945 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:17 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:21:17.945 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1994510970' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi094-78099-66"}]': finished 2024-03-12T03:21:18.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:21:17 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:21:18.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:18 smithi094 ceph-mon[29291]: pgmap v506: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:21:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:18 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-51", "mode": "writeback"}]': finished 2024-03-12T03:21:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:18 smithi094 ceph-mon[29291]: osdmap e366: 8 total, 8 up, 8 in 2024-03-12T03:21:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:21:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:18 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:21:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:18 smithi094 ceph-mon[40484]: pgmap v506: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:21:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:18 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-51", "mode": "writeback"}]': finished 2024-03-12T03:21:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:18 smithi094 ceph-mon[40484]: osdmap e366: 8 total, 8 up, 8 in 2024-03-12T03:21:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:21:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:18 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:21:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:18 smithi151 ceph-mon[32923]: pgmap v506: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:21:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:18 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-51", "mode": "writeback"}]': finished 2024-03-12T03:21:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:18 smithi151 ceph-mon[32923]: osdmap e366: 8 total, 8 up, 8 in 2024-03-12T03:21:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:21:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:18 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:21:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:19 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:21:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:19 smithi094 ceph-mon[40484]: osdmap e367: 8 total, 8 up, 8 in 2024-03-12T03:21:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:21:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:19 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:21:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:19.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:19.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:19 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:21:19.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:19 smithi094 ceph-mon[29291]: osdmap e367: 8 total, 8 up, 8 in 2024-03-12T03:21:19.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:21:19.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:19 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:21:19.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:19 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:21:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:19 smithi151 ceph-mon[32923]: osdmap e367: 8 total, 8 up, 8 in 2024-03-12T03:21:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:21:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:19 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:21:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:20.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:20 smithi094 ceph-mon[40484]: pgmap v509: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-12T03:21:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:20 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:21:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:21:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:20 smithi094 ceph-mon[40484]: osdmap e368: 8 total, 8 up, 8 in 2024-03-12T03:21:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:20 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:21:20.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:21:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:21:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:21:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:20 smithi094 ceph-mon[29291]: pgmap v509: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-12T03:21:20.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:20 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:21:20.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:21:20.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:20 smithi094 ceph-mon[29291]: osdmap e368: 8 total, 8 up, 8 in 2024-03-12T03:21:20.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:20 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:21:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:20 smithi151 ceph-mon[32923]: pgmap v509: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-12T03:21:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:20 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:21:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:21:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:20 smithi151 ceph-mon[32923]: osdmap e368: 8 total, 8 up, 8 in 2024-03-12T03:21:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:20 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:21:21.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:21:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:21:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[29291]: osdmap e369: 8 total, 8 up, 8 in 2024-03-12T03:21:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:21:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:21:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[29291]: pgmap v512: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-12T03:21:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-12T03:21:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:21:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:21:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[40484]: osdmap e369: 8 total, 8 up, 8 in 2024-03-12T03:21:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:21:21.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:21:21.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[40484]: pgmap v512: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-12T03:21:21.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:21 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-12T03:21:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:21 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:21:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:21 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:21:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:21 smithi151 ceph-mon[32923]: osdmap e369: 8 total, 8 up, 8 in 2024-03-12T03:21:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:21:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:21 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:21:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:21 smithi151 ceph-mon[32923]: pgmap v512: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-12T03:21:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:21 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-12T03:21:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:22 smithi094 ceph-mon[29291]: osdmap e370: 8 total, 8 up, 8 in 2024-03-12T03:21:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-12T03:21:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:22 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-12T03:21:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:22 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "target_max_objects","val": "1"}]': finished 2024-03-12T03:21:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:22 smithi094 ceph-mon[29291]: osdmap e371: 8 total, 8 up, 8 in 2024-03-12T03:21:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:22 smithi094 ceph-mon[40484]: osdmap e370: 8 total, 8 up, 8 in 2024-03-12T03:21:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-12T03:21:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:22 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-12T03:21:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:22 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "target_max_objects","val": "1"}]': finished 2024-03-12T03:21:22.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:22 smithi094 ceph-mon[40484]: osdmap e371: 8 total, 8 up, 8 in 2024-03-12T03:21:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:22 smithi151 ceph-mon[32923]: osdmap e370: 8 total, 8 up, 8 in 2024-03-12T03:21:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-12T03:21:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:22 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-12T03:21:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:22 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-51","var": "target_max_objects","val": "1"}]': finished 2024-03-12T03:21:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:22 smithi151 ceph-mon[32923]: osdmap e371: 8 total, 8 up, 8 in 2024-03-12T03:21:23.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:23 smithi094 ceph-mon[29291]: pgmap v515: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:23.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:23 smithi094 ceph-mon[40484]: pgmap v515: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:23.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:23 smithi151 ceph-mon[32923]: pgmap v515: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:24.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:24 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-12T03:21:24.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:24 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-12T03:21:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:24 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-12T03:21:25.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:25 smithi094 ceph-mon[29291]: pgmap v516: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-03-12T03:21:25.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:25 smithi094 ceph-mon[40484]: pgmap v516: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-03-12T03:21:25.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:25 smithi151 ceph-mon[32923]: pgmap v516: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-03-12T03:21:27.955 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:27 smithi151 ceph-mon[32923]: pgmap v517: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-12T03:21:27.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:27 smithi094 ceph-mon[29291]: pgmap v517: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-12T03:21:27.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:27 smithi094 ceph-mon[40484]: pgmap v517: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-12T03:21:28.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:21:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:21:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:28 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:28 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:28 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:29.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:29 smithi094 ceph-mon[29291]: pgmap v518: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:21:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:29 smithi094 ceph-mon[40484]: pgmap v518: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:21:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:29 smithi151 ceph-mon[32923]: pgmap v518: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:21:30.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:21:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:21:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:21:31.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:31 smithi094 ceph-mon[29291]: pgmap v519: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:21:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:31 smithi094 ceph-mon[40484]: pgmap v519: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:21:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:31 smithi151 ceph-mon[32923]: pgmap v519: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:21:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:32 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:33.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:33.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:32 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:33.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:33.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:32 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:34.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:33 smithi094 ceph-mon[29291]: pgmap v520: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:21:34.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:33 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:21:34.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:33 smithi094 ceph-mon[29291]: osdmap e372: 8 total, 8 up, 8 in 2024-03-12T03:21:34.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:34.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:33 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:34.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:33 smithi094 ceph-mon[40484]: pgmap v520: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:21:34.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:33 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:21:34.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:33 smithi094 ceph-mon[40484]: osdmap e372: 8 total, 8 up, 8 in 2024-03-12T03:21:34.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:34.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:33 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:34.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:33 smithi151 ceph-mon[32923]: pgmap v520: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:21:34.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:33 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:21:34.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:33 smithi151 ceph-mon[32923]: osdmap e372: 8 total, 8 up, 8 in 2024-03-12T03:21:34.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:34.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:33 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]': finished 2024-03-12T03:21:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[29291]: osdmap e373: 8 total, 8 up, 8 in 2024-03-12T03:21:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:35.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:35.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:21:35.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:35.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]': finished 2024-03-12T03:21:35.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[40484]: osdmap e373: 8 total, 8 up, 8 in 2024-03-12T03:21:35.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:35.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:35.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:35.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:35.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:21:35.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:35.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:34 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]': finished 2024-03-12T03:21:35.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:34 smithi151 ceph-mon[32923]: osdmap e373: 8 total, 8 up, 8 in 2024-03-12T03:21:35.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:35.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:34 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:35.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:35.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:34 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-51"}]: dispatch 2024-03-12T03:21:35.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:34 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:21:35.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:36.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:35 smithi094 ceph-mon[29291]: pgmap v523: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:21:36.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:35 smithi094 ceph-mon[29291]: osdmap e374: 8 total, 8 up, 8 in 2024-03-12T03:21:36.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:35 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-12T03:21:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:35 smithi094 ceph-mon[40484]: pgmap v523: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:21:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:35 smithi094 ceph-mon[40484]: osdmap e374: 8 total, 8 up, 8 in 2024-03-12T03:21:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:35 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-12T03:21:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:35 smithi151 ceph-mon[32923]: pgmap v523: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:21:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:35 smithi151 ceph-mon[32923]: osdmap e374: 8 total, 8 up, 8 in 2024-03-12T03:21:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:35 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-12T03:21:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:36 smithi094 ceph-mon[40484]: osdmap e375: 8 total, 8 up, 8 in 2024-03-12T03:21:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:36 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:37.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:36 smithi094 ceph-mon[29291]: osdmap e375: 8 total, 8 up, 8 in 2024-03-12T03:21:37.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:37.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:36 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:36 smithi151 ceph-mon[32923]: osdmap e375: 8 total, 8 up, 8 in 2024-03-12T03:21:37.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:37.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:36 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[40484]: pgmap v526: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:21:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[40484]: osdmap e376: 8 total, 8 up, 8 in 2024-03-12T03:21:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-53"}]: dispatch 2024-03-12T03:21:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-53"}]: dispatch 2024-03-12T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[29291]: pgmap v526: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[29291]: osdmap e376: 8 total, 8 up, 8 in 2024-03-12T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-53"}]: dispatch 2024-03-12T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:37 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-53"}]: dispatch 2024-03-12T03:21:38.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:21:37 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:21:38.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:37 smithi151 ceph-mon[32923]: pgmap v526: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:21:38.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:37 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:38.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:37 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:38.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:37 smithi151 ceph-mon[32923]: osdmap e376: 8 total, 8 up, 8 in 2024-03-12T03:21:38.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:38.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:37 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:38.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-53"}]: dispatch 2024-03-12T03:21:38.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:37 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-53"}]: dispatch 2024-03-12T03:21:39.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:38 smithi094 ceph-mon[29291]: osdmap e377: 8 total, 8 up, 8 in 2024-03-12T03:21:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:38 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:38 smithi094 ceph-mon[40484]: osdmap e377: 8 total, 8 up, 8 in 2024-03-12T03:21:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:38 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:39.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:38 smithi151 ceph-mon[32923]: osdmap e377: 8 total, 8 up, 8 in 2024-03-12T03:21:39.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:38 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:40.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:39 smithi094 ceph-mon[29291]: pgmap v529: 260 pgs: 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:39 smithi094 ceph-mon[29291]: osdmap e378: 8 total, 8 up, 8 in 2024-03-12T03:21:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:39 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:40.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:39 smithi094 ceph-mon[40484]: pgmap v529: 260 pgs: 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:40.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:39 smithi094 ceph-mon[40484]: osdmap e378: 8 total, 8 up, 8 in 2024-03-12T03:21:40.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:40.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:39 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:40.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:39 smithi151 ceph-mon[32923]: pgmap v529: 260 pgs: 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:40.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:39 smithi151 ceph-mon[32923]: osdmap e378: 8 total, 8 up, 8 in 2024-03-12T03:21:40.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:40.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:39 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:40.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:21:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:21:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[29291]: osdmap e379: 8 total, 8 up, 8 in 2024-03-12T03:21:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:21:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-55"}]: dispatch 2024-03-12T03:21:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-55"}]: dispatch 2024-03-12T03:21:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[40484]: osdmap e379: 8 total, 8 up, 8 in 2024-03-12T03:21:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:21:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-55"}]: dispatch 2024-03-12T03:21:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:40 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-55"}]: dispatch 2024-03-12T03:21:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:40 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:41.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:40 smithi151 ceph-mon[32923]: osdmap e379: 8 total, 8 up, 8 in 2024-03-12T03:21:41.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:21:41.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:41.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:40 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:41.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-55"}]: dispatch 2024-03-12T03:21:41.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:40 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-55"}]: dispatch 2024-03-12T03:21:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:41 smithi094 ceph-mon[29291]: pgmap v532: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:21:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:41 smithi094 ceph-mon[29291]: osdmap e380: 8 total, 8 up, 8 in 2024-03-12T03:21:42.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:41 smithi094 ceph-mon[40484]: pgmap v532: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:21:42.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:41 smithi094 ceph-mon[40484]: osdmap e380: 8 total, 8 up, 8 in 2024-03-12T03:21:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:41 smithi151 ceph-mon[32923]: pgmap v532: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:21:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:41 smithi151 ceph-mon[32923]: osdmap e380: 8 total, 8 up, 8 in 2024-03-12T03:21:43.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:42 smithi094 ceph-mon[29291]: osdmap e381: 8 total, 8 up, 8 in 2024-03-12T03:21:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:42 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:42 smithi094 ceph-mon[40484]: osdmap e381: 8 total, 8 up, 8 in 2024-03-12T03:21:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:42 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:42 smithi151 ceph-mon[32923]: osdmap e381: 8 total, 8 up, 8 in 2024-03-12T03:21:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:42 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:44.115 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:43 smithi151 ceph-mon[32923]: pgmap v535: 292 pgs: 5 creating+peering, 27 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:43 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:43 smithi151 ceph-mon[32923]: osdmap e382: 8 total, 8 up, 8 in 2024-03-12T03:21:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-57"}]: dispatch 2024-03-12T03:21:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-57"}]: dispatch 2024-03-12T03:21:44.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[29291]: pgmap v535: 292 pgs: 5 creating+peering, 27 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[29291]: osdmap e382: 8 total, 8 up, 8 in 2024-03-12T03:21:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-57"}]: dispatch 2024-03-12T03:21:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-57"}]: dispatch 2024-03-12T03:21:44.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[40484]: pgmap v535: 292 pgs: 5 creating+peering, 27 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:44.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:44.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:44.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[40484]: osdmap e382: 8 total, 8 up, 8 in 2024-03-12T03:21:44.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:44.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:44.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-57"}]: dispatch 2024-03-12T03:21:44.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-57"}]: dispatch 2024-03-12T03:21:45.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:44 smithi094 ceph-mon[29291]: osdmap e383: 8 total, 8 up, 8 in 2024-03-12T03:21:45.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:44 smithi094 ceph-mon[40484]: osdmap e383: 8 total, 8 up, 8 in 2024-03-12T03:21:45.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:44 smithi151 ceph-mon[32923]: osdmap e383: 8 total, 8 up, 8 in 2024-03-12T03:21:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:45 smithi094 ceph-mon[29291]: pgmap v538: 260 pgs: 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:21:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:45 smithi094 ceph-mon[29291]: osdmap e384: 8 total, 8 up, 8 in 2024-03-12T03:21:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:45 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:46.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:45 smithi094 ceph-mon[40484]: pgmap v538: 260 pgs: 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:21:46.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:45 smithi094 ceph-mon[40484]: osdmap e384: 8 total, 8 up, 8 in 2024-03-12T03:21:46.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:46.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:45 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:46.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:45 smithi151 ceph-mon[32923]: pgmap v538: 260 pgs: 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:21:46.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:45 smithi151 ceph-mon[32923]: osdmap e384: 8 total, 8 up, 8 in 2024-03-12T03:21:46.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:46.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:45 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:47.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:47.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[29291]: osdmap e385: 8 total, 8 up, 8 in 2024-03-12T03:21:47.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:21:47.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:47.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:47.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-59"}]: dispatch 2024-03-12T03:21:47.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-59"}]: dispatch 2024-03-12T03:21:47.112 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:47.112 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[40484]: osdmap e385: 8 total, 8 up, 8 in 2024-03-12T03:21:47.112 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:21:47.112 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:47.112 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:47.112 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-59"}]: dispatch 2024-03-12T03:21:47.112 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:46 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-59"}]: dispatch 2024-03-12T03:21:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:46 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:47.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:46 smithi151 ceph-mon[32923]: osdmap e385: 8 total, 8 up, 8 in 2024-03-12T03:21:47.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:21:47.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:47.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:46 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:47.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-59"}]: dispatch 2024-03-12T03:21:47.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:46 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-59"}]: dispatch 2024-03-12T03:21:48.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:47 smithi094 ceph-mon[29291]: pgmap v541: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:21:48.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:47 smithi094 ceph-mon[29291]: osdmap e386: 8 total, 8 up, 8 in 2024-03-12T03:21:48.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:47 smithi094 ceph-mon[40484]: pgmap v541: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:21:48.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:47 smithi094 ceph-mon[40484]: osdmap e386: 8 total, 8 up, 8 in 2024-03-12T03:21:48.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:47 smithi151 ceph-mon[32923]: pgmap v541: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:21:48.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:47 smithi151 ceph-mon[32923]: osdmap e386: 8 total, 8 up, 8 in 2024-03-12T03:21:48.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:21:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:21:49.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[29291]: osdmap e387: 8 total, 8 up, 8 in 2024-03-12T03:21:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[29291]: osdmap e388: 8 total, 8 up, 8 in 2024-03-12T03:21:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[40484]: osdmap e387: 8 total, 8 up, 8 in 2024-03-12T03:21:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:49.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:49.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:49.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:49.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:49.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:48 smithi094 ceph-mon[40484]: osdmap e388: 8 total, 8 up, 8 in 2024-03-12T03:21:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:48 smithi151 ceph-mon[32923]: osdmap e387: 8 total, 8 up, 8 in 2024-03-12T03:21:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:48 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:49.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:48 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:49.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:48 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:49.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:48 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:49.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:48 smithi151 ceph-mon[32923]: osdmap e388: 8 total, 8 up, 8 in 2024-03-12T03:21:50.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[29291]: pgmap v544: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-12T03:21:50.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:21:50.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:21:50.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:21:50.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:21:50.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:50.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:21:50.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[29291]: osdmap e389: 8 total, 8 up, 8 in 2024-03-12T03:21:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[40484]: pgmap v544: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-12T03:21:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:21:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:21:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:21:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:21:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:21:50.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:49 smithi094 ceph-mon[40484]: osdmap e389: 8 total, 8 up, 8 in 2024-03-12T03:21:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:49 smithi151 ceph-mon[32923]: pgmap v544: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-12T03:21:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:21:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:21:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:21:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:49 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:21:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:49 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:21:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:49 smithi151 ceph-mon[32923]: osdmap e389: 8 total, 8 up, 8 in 2024-03-12T03:21:50.839 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:21:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:21:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:21:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:50 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-61"}]: dispatch 2024-03-12T03:21:51.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:50 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-61"}]: dispatch 2024-03-12T03:21:51.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:51.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:50 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:51.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-61"}]: dispatch 2024-03-12T03:21:51.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:50 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-61"}]: dispatch 2024-03-12T03:21:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:50 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:21:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-61"}]: dispatch 2024-03-12T03:21:51.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:50 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-61"}]: dispatch 2024-03-12T03:21:52.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:51 smithi094 ceph-mon[29291]: pgmap v547: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-12T03:21:52.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:51 smithi094 ceph-mon[29291]: osdmap e390: 8 total, 8 up, 8 in 2024-03-12T03:21:52.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:51 smithi094 ceph-mon[40484]: pgmap v547: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-12T03:21:52.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:51 smithi094 ceph-mon[40484]: osdmap e390: 8 total, 8 up, 8 in 2024-03-12T03:21:52.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:51 smithi151 ceph-mon[32923]: pgmap v547: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-12T03:21:52.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:51 smithi151 ceph-mon[32923]: osdmap e390: 8 total, 8 up, 8 in 2024-03-12T03:21:53.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:52 smithi094 ceph-mon[29291]: osdmap e391: 8 total, 8 up, 8 in 2024-03-12T03:21:53.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:53.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:53.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:52 smithi094 ceph-mon[40484]: osdmap e391: 8 total, 8 up, 8 in 2024-03-12T03:21:53.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:53.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:53.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:52 smithi151 ceph-mon[32923]: osdmap e391: 8 total, 8 up, 8 in 2024-03-12T03:21:53.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:53.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:54.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[29291]: pgmap v550: 292 pgs: 3 creating+peering, 29 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:54.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:54.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[29291]: osdmap e392: 8 total, 8 up, 8 in 2024-03-12T03:21:54.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:21:54.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:21:54.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:21:54.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:54.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[40484]: pgmap v550: 292 pgs: 3 creating+peering, 29 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:54.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:54.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[40484]: osdmap e392: 8 total, 8 up, 8 in 2024-03-12T03:21:54.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:21:54.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:21:54.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:21:54.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:53 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:54.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:53 smithi151 ceph-mon[32923]: pgmap v550: 292 pgs: 3 creating+peering, 29 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:21:54.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:53 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:54.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:53 smithi151 ceph-mon[32923]: osdmap e392: 8 total, 8 up, 8 in 2024-03-12T03:21:54.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:21:54.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:21:54.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:53 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:21:54.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:53 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:55.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:54 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:21:55.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:54 smithi094 ceph-mon[29291]: osdmap e393: 8 total, 8 up, 8 in 2024-03-12T03:21:55.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:54 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:21:55.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:54 smithi094 ceph-mon[40484]: osdmap e393: 8 total, 8 up, 8 in 2024-03-12T03:21:55.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:54 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:21:55.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:54 smithi151 ceph-mon[32923]: osdmap e393: 8 total, 8 up, 8 in 2024-03-12T03:21:56.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:55 smithi094 ceph-mon[29291]: pgmap v553: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:21:56.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:55 smithi094 ceph-mon[29291]: osdmap e394: 8 total, 8 up, 8 in 2024-03-12T03:21:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:55 smithi094 ceph-mon[40484]: pgmap v553: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:21:56.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:55 smithi094 ceph-mon[40484]: osdmap e394: 8 total, 8 up, 8 in 2024-03-12T03:21:56.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:55 smithi151 ceph-mon[32923]: pgmap v553: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:21:56.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:55 smithi151 ceph-mon[32923]: osdmap e394: 8 total, 8 up, 8 in 2024-03-12T03:21:57.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:56 smithi094 ceph-mon[29291]: osdmap e395: 8 total, 8 up, 8 in 2024-03-12T03:21:57.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:56 smithi094 ceph-mon[40484]: osdmap e395: 8 total, 8 up, 8 in 2024-03-12T03:21:57.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:56 smithi151 ceph-mon[32923]: osdmap e395: 8 total, 8 up, 8 in 2024-03-12T03:21:58.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:57 smithi094 ceph-mon[29291]: pgmap v556: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:21:58.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:57 smithi094 ceph-mon[29291]: osdmap e396: 8 total, 8 up, 8 in 2024-03-12T03:21:58.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:57 smithi094 ceph-mon[40484]: pgmap v556: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:21:58.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:57 smithi094 ceph-mon[40484]: osdmap e396: 8 total, 8 up, 8 in 2024-03-12T03:21:58.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:57 smithi151 ceph-mon[32923]: pgmap v556: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:21:58.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:57 smithi151 ceph-mon[32923]: osdmap e396: 8 total, 8 up, 8 in 2024-03-12T03:21:58.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:21:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:21:59.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:58 smithi094 ceph-mon[29291]: osdmap e397: 8 total, 8 up, 8 in 2024-03-12T03:21:59.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:58 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:59.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:58 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:59.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:58 smithi094 ceph-mon[40484]: osdmap e397: 8 total, 8 up, 8 in 2024-03-12T03:21:59.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:58 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:59.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:58 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:21:59.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:58 smithi151 ceph-mon[32923]: osdmap e397: 8 total, 8 up, 8 in 2024-03-12T03:21:59.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:58 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:21:59.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:58 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:00.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:21:59 smithi094 ceph-mon[29291]: pgmap v559: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-03-12T03:22:00.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:21:59 smithi094 ceph-mon[40484]: pgmap v559: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-03-12T03:22:00.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:21:59 smithi151 ceph-mon[32923]: pgmap v559: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-03-12T03:22:00.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:22:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:22:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:22:02.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:01 smithi094 ceph-mon[40484]: pgmap v560: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-03-12T03:22:02.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:01 smithi094 ceph-mon[29291]: pgmap v560: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-03-12T03:22:02.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:01 smithi151 ceph-mon[32923]: pgmap v560: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-03-12T03:22:04.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:03 smithi094 ceph-mon[40484]: pgmap v561: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-12T03:22:04.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:03 smithi094 ceph-mon[29291]: pgmap v561: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-12T03:22:04.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:03 smithi151 ceph-mon[32923]: pgmap v561: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-12T03:22:05.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:05.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:05.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:05 smithi094 ceph-mon[29291]: pgmap v562: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-12T03:22:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:05 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:22:06.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:05 smithi094 ceph-mon[40484]: pgmap v562: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-12T03:22:06.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:06.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:06.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:06.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:05 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:22:06.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:05 smithi151 ceph-mon[32923]: pgmap v562: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-12T03:22:06.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:06.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:06.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:06.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:05 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:22:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:07 smithi094 ceph-mon[29291]: pgmap v563: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-12T03:22:08.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:07 smithi094 ceph-mon[40484]: pgmap v563: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-12T03:22:08.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:07 smithi151 ceph-mon[32923]: pgmap v563: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-12T03:22:08.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:22:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:22:09.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:08 smithi094 ceph-mon[29291]: osdmap e398: 8 total, 8 up, 8 in 2024-03-12T03:22:09.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:08 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:09.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:08 smithi094 ceph-mon[29291]: osdmap e399: 8 total, 8 up, 8 in 2024-03-12T03:22:09.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:08 smithi094 ceph-mon[40484]: osdmap e398: 8 total, 8 up, 8 in 2024-03-12T03:22:09.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:08 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:09.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:08 smithi094 ceph-mon[40484]: osdmap e399: 8 total, 8 up, 8 in 2024-03-12T03:22:09.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:08 smithi151 ceph-mon[32923]: osdmap e398: 8 total, 8 up, 8 in 2024-03-12T03:22:09.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:08 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:09.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:08 smithi151 ceph-mon[32923]: osdmap e399: 8 total, 8 up, 8 in 2024-03-12T03:22:10.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:10 smithi094 ceph-mon[40484]: pgmap v566: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:10 smithi094 ceph-mon[29291]: pgmap v566: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:10.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:10 smithi151 ceph-mon[32923]: pgmap v566: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:10.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:22:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:22:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:22:12.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:12 smithi094 ceph-mon[29291]: pgmap v567: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:22:12.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:12 smithi094 ceph-mon[40484]: pgmap v567: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:22:12.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:12 smithi151 ceph-mon[32923]: pgmap v567: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:22:14.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:14 smithi094 ceph-mon[29291]: pgmap v568: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:14.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:14 smithi094 ceph-mon[40484]: pgmap v568: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:14.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:14 smithi151 ceph-mon[32923]: pgmap v568: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:16.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:16 smithi094 ceph-mon[29291]: pgmap v569: 292 pgs: 292 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:16.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:16 smithi094 ceph-mon[40484]: pgmap v569: 292 pgs: 292 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:16.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:16 smithi151 ceph-mon[32923]: pgmap v569: 292 pgs: 292 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-03-12T03:22:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.e", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-03-12T03:22:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "106.1a", "id": [2, 5]}]: dispatch 2024-03-12T03:22:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.e", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "106.1a", "id": [2, 5]}]: dispatch 2024-03-12T03:22:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-03-12T03:22:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.e", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-03-12T03:22:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "106.1a", "id": [2, 5]}]: dispatch 2024-03-12T03:22:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.e", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "106.1a", "id": [2, 5]}]: dispatch 2024-03-12T03:22:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-03-12T03:22:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.e", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-03-12T03:22:17.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "106.1a", "id": [2, 5]}]: dispatch 2024-03-12T03:22:17.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.e", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]: dispatch 2024-03-12T03:22:17.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "106.1a", "id": [2, 5]}]: dispatch 2024-03-12T03:22:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[29291]: pgmap v570: 292 pgs: 292 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]': finished 2024-03-12T03:22:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]': finished 2024-03-12T03:22:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.e", "id": [3, 1]}]': finished 2024-03-12T03:22:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]': finished 2024-03-12T03:22:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "106.1a", "id": [2, 5]}]': finished 2024-03-12T03:22:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[29291]: osdmap e400: 8 total, 8 up, 8 in 2024-03-12T03:22:18.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[29291]: osdmap e401: 8 total, 8 up, 8 in 2024-03-12T03:22:18.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[40484]: pgmap v570: 292 pgs: 292 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]': finished 2024-03-12T03:22:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]': finished 2024-03-12T03:22:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.e", "id": [3, 1]}]': finished 2024-03-12T03:22:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]': finished 2024-03-12T03:22:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "106.1a", "id": [2, 5]}]': finished 2024-03-12T03:22:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[40484]: osdmap e400: 8 total, 8 up, 8 in 2024-03-12T03:22:18.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:18 smithi094 ceph-mon[40484]: osdmap e401: 8 total, 8 up, 8 in 2024-03-12T03:22:18.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:22:17 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:22:18.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:18 smithi151 ceph-mon[32923]: pgmap v570: 292 pgs: 292 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:18.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:18 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]': finished 2024-03-12T03:22:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:18 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]': finished 2024-03-12T03:22:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:18 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.e", "id": [3, 1]}]': finished 2024-03-12T03:22:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:18 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]': finished 2024-03-12T03:22:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:18 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "106.1a", "id": [2, 5]}]': finished 2024-03-12T03:22:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:18 smithi151 ceph-mon[32923]: osdmap e400: 8 total, 8 up, 8 in 2024-03-12T03:22:18.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:18 smithi151 ceph-mon[32923]: osdmap e401: 8 total, 8 up, 8 in 2024-03-12T03:22:19.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:19.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:19.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:20.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:20 smithi094 ceph-mon[29291]: pgmap v573: 292 pgs: 5 peering, 287 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:20.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:20 smithi094 ceph-mon[29291]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-12T03:22:20.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:20 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:22:20.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:20.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:20 smithi094 ceph-mon[40484]: pgmap v573: 292 pgs: 5 peering, 287 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:20.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:20 smithi094 ceph-mon[40484]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-12T03:22:20.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:20 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:22:20.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:20.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:20 smithi151 ceph-mon[32923]: pgmap v573: 292 pgs: 5 peering, 287 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:22:20.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:20 smithi151 ceph-mon[32923]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-12T03:22:20.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:20 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:22:20.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:20.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:22:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:22:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:22:22.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:22 smithi151 ceph-mon[32923]: pgmap v574: 292 pgs: 5 peering, 287 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 0 op/s 2024-03-12T03:22:22.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:22 smithi094 ceph-mon[29291]: pgmap v574: 292 pgs: 5 peering, 287 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 0 op/s 2024-03-12T03:22:22.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:22 smithi094 ceph-mon[40484]: pgmap v574: 292 pgs: 5 peering, 287 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 0 op/s 2024-03-12T03:22:24.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:24 smithi094 ceph-mon[29291]: pgmap v575: 292 pgs: 4 peering, 288 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:22:24.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:24 smithi094 ceph-mon[40484]: pgmap v575: 292 pgs: 4 peering, 288 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:22:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:24 smithi151 ceph-mon[32923]: pgmap v575: 292 pgs: 4 peering, 288 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:22:25.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:25 smithi094 ceph-mon[29291]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-12T03:22:25.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:25 smithi094 ceph-mon[40484]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-12T03:22:25.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:25 smithi151 ceph-mon[32923]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-12T03:22:26.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:26 smithi094 ceph-mon[29291]: pgmap v576: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 5 B/s, 2 objects/s recovering 2024-03-12T03:22:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:26 smithi094 ceph-mon[40484]: pgmap v576: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 5 B/s, 2 objects/s recovering 2024-03-12T03:22:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:26 smithi151 ceph-mon[32923]: pgmap v576: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 5 B/s, 2 objects/s recovering 2024-03-12T03:22:28.271 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:22:27 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:22:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:28 smithi094 ceph-mon[29291]: pgmap v577: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4 B/s, 2 objects/s recovering 2024-03-12T03:22:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:28 smithi094 ceph-mon[29291]: osdmap e402: 8 total, 8 up, 8 in 2024-03-12T03:22:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:28 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:28.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-63"}]: dispatch 2024-03-12T03:22:28.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:28 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-63"}]: dispatch 2024-03-12T03:22:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:28 smithi094 ceph-mon[40484]: pgmap v577: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4 B/s, 2 objects/s recovering 2024-03-12T03:22:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:28 smithi094 ceph-mon[40484]: osdmap e402: 8 total, 8 up, 8 in 2024-03-12T03:22:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:28 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-63"}]: dispatch 2024-03-12T03:22:28.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:28 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-63"}]: dispatch 2024-03-12T03:22:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:28 smithi151 ceph-mon[32923]: pgmap v577: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4 B/s, 2 objects/s recovering 2024-03-12T03:22:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:28 smithi151 ceph-mon[32923]: osdmap e402: 8 total, 8 up, 8 in 2024-03-12T03:22:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:28 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-63"}]: dispatch 2024-03-12T03:22:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:28 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-63"}]: dispatch 2024-03-12T03:22:29.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:29 smithi094 ceph-mon[29291]: osdmap e403: 8 total, 8 up, 8 in 2024-03-12T03:22:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:29 smithi094 ceph-mon[40484]: osdmap e403: 8 total, 8 up, 8 in 2024-03-12T03:22:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:29 smithi151 ceph-mon[32923]: osdmap e403: 8 total, 8 up, 8 in 2024-03-12T03:22:30.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:22:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:22:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:22:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:30 smithi094 ceph-mon[29291]: pgmap v579: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 4 B/s, 1 objects/s recovering 2024-03-12T03:22:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:30 smithi094 ceph-mon[29291]: osdmap e404: 8 total, 8 up, 8 in 2024-03-12T03:22:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:30 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:30.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:30 smithi094 ceph-mon[40484]: pgmap v579: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 4 B/s, 1 objects/s recovering 2024-03-12T03:22:30.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:30 smithi094 ceph-mon[40484]: osdmap e404: 8 total, 8 up, 8 in 2024-03-12T03:22:30.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:30.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:30 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:30 smithi151 ceph-mon[32923]: pgmap v579: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 4 B/s, 1 objects/s recovering 2024-03-12T03:22:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:30 smithi151 ceph-mon[32923]: osdmap e404: 8 total, 8 up, 8 in 2024-03-12T03:22:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:30 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[29291]: pgmap v582: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:22:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[29291]: osdmap e405: 8 total, 8 up, 8 in 2024-03-12T03:22:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:22:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:22:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:22:32.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:32.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[40484]: pgmap v582: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:22:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[40484]: osdmap e405: 8 total, 8 up, 8 in 2024-03-12T03:22:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:22:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:22:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:22:32.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:32 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:32.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:32 smithi151 ceph-mon[32923]: pgmap v582: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:32.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:32 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:22:32.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:32 smithi151 ceph-mon[32923]: osdmap e405: 8 total, 8 up, 8 in 2024-03-12T03:22:32.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:22:32.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:22:32.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:32 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:22:32.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:32 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:33.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:33 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:22:33.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:33 smithi151 ceph-mon[32923]: osdmap e406: 8 total, 8 up, 8 in 2024-03-12T03:22:33.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:33 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:22:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:33 smithi094 ceph-mon[29291]: osdmap e406: 8 total, 8 up, 8 in 2024-03-12T03:22:33.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:33 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:22:33.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:33 smithi094 ceph-mon[40484]: osdmap e406: 8 total, 8 up, 8 in 2024-03-12T03:22:34.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:34 smithi094 ceph-mon[29291]: pgmap v585: 292 pgs: 18 unknown, 274 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:34 smithi094 ceph-mon[29291]: osdmap e407: 8 total, 8 up, 8 in 2024-03-12T03:22:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:34 smithi094 ceph-mon[40484]: pgmap v585: 292 pgs: 18 unknown, 274 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:34 smithi094 ceph-mon[40484]: osdmap e407: 8 total, 8 up, 8 in 2024-03-12T03:22:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:34 smithi151 ceph-mon[32923]: pgmap v585: 292 pgs: 18 unknown, 274 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:34 smithi151 ceph-mon[32923]: osdmap e407: 8 total, 8 up, 8 in 2024-03-12T03:22:35.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:35 smithi094 ceph-mon[29291]: osdmap e408: 8 total, 8 up, 8 in 2024-03-12T03:22:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:35 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:22:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:35 smithi094 ceph-mon[40484]: osdmap e408: 8 total, 8 up, 8 in 2024-03-12T03:22:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:35 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:22:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:35 smithi151 ceph-mon[32923]: osdmap e408: 8 total, 8 up, 8 in 2024-03-12T03:22:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:35 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:22:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:36.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:36 smithi094 ceph-mon[29291]: pgmap v588: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-12T03:22:36.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:36 smithi094 ceph-mon[29291]: osdmap e409: 8 total, 8 up, 8 in 2024-03-12T03:22:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:36 smithi094 ceph-mon[40484]: pgmap v588: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-12T03:22:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:36 smithi094 ceph-mon[40484]: osdmap e409: 8 total, 8 up, 8 in 2024-03-12T03:22:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:36 smithi151 ceph-mon[32923]: pgmap v588: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-12T03:22:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:36 smithi151 ceph-mon[32923]: osdmap e409: 8 total, 8 up, 8 in 2024-03-12T03:22:38.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:22:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:22:38.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:38 smithi151 ceph-mon[32923]: pgmap v590: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-12T03:22:38.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:38 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:38 smithi094 ceph-mon[29291]: pgmap v590: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-12T03:22:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:38 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:38 smithi094 ceph-mon[40484]: pgmap v590: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-12T03:22:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:38 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:39.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:40.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:40 smithi094 ceph-mon[29291]: pgmap v591: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-12T03:22:40.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:40 smithi094 ceph-mon[40484]: pgmap v591: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-12T03:22:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:40 smithi151 ceph-mon[32923]: pgmap v591: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-12T03:22:40.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:22:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:22:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:22:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:42 smithi094 ceph-mon[29291]: pgmap v592: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 549 B/s wr, 3 op/s 2024-03-12T03:22:42.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:42 smithi094 ceph-mon[40484]: pgmap v592: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 549 B/s wr, 3 op/s 2024-03-12T03:22:42.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:42 smithi151 ceph-mon[32923]: pgmap v592: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 549 B/s wr, 3 op/s 2024-03-12T03:22:44.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:44 smithi094 ceph-mon[29291]: pgmap v593: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 484 B/s wr, 3 op/s 2024-03-12T03:22:44.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:44 smithi094 ceph-mon[29291]: osdmap e410: 8 total, 8 up, 8 in 2024-03-12T03:22:44.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:44 smithi094 ceph-mon[40484]: pgmap v593: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 484 B/s wr, 3 op/s 2024-03-12T03:22:44.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:44 smithi094 ceph-mon[40484]: osdmap e410: 8 total, 8 up, 8 in 2024-03-12T03:22:44.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:44 smithi151 ceph-mon[32923]: pgmap v593: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 484 B/s wr, 3 op/s 2024-03-12T03:22:44.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:44 smithi151 ceph-mon[32923]: osdmap e410: 8 total, 8 up, 8 in 2024-03-12T03:22:46.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:46 smithi094 ceph-mon[29291]: pgmap v595: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 433 B/s wr, 3 op/s 2024-03-12T03:22:46.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:46.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:46 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:46.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-65"}]: dispatch 2024-03-12T03:22:46.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:46 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-65"}]: dispatch 2024-03-12T03:22:46.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:46 smithi094 ceph-mon[40484]: pgmap v595: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 433 B/s wr, 3 op/s 2024-03-12T03:22:46.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:46.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:46 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:46.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-65"}]: dispatch 2024-03-12T03:22:46.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:46 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-65"}]: dispatch 2024-03-12T03:22:46.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:46 smithi151 ceph-mon[32923]: pgmap v595: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 433 B/s wr, 3 op/s 2024-03-12T03:22:46.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:46.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:46 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:46.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-65"}]: dispatch 2024-03-12T03:22:46.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:46 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-65"}]: dispatch 2024-03-12T03:22:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:47 smithi094 ceph-mon[29291]: osdmap e411: 8 total, 8 up, 8 in 2024-03-12T03:22:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:47 smithi094 ceph-mon[40484]: osdmap e411: 8 total, 8 up, 8 in 2024-03-12T03:22:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:47 smithi151 ceph-mon[32923]: osdmap e411: 8 total, 8 up, 8 in 2024-03-12T03:22:48.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:22:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:22:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:48 smithi151 ceph-mon[32923]: pgmap v597: 260 pgs: 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:48 smithi151 ceph-mon[32923]: osdmap e412: 8 total, 8 up, 8 in 2024-03-12T03:22:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:48 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:48 smithi094 ceph-mon[40484]: pgmap v597: 260 pgs: 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:48 smithi094 ceph-mon[40484]: osdmap e412: 8 total, 8 up, 8 in 2024-03-12T03:22:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:48 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:48 smithi094 ceph-mon[29291]: pgmap v597: 260 pgs: 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:48 smithi094 ceph-mon[29291]: osdmap e412: 8 total, 8 up, 8 in 2024-03-12T03:22:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:48 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:22:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:49 smithi094 ceph-mon[29291]: osdmap e413: 8 total, 8 up, 8 in 2024-03-12T03:22:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:22:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:49 smithi094 ceph-mon[29291]: osdmap e414: 8 total, 8 up, 8 in 2024-03-12T03:22:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:22:49.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:49 smithi094 ceph-mon[40484]: osdmap e413: 8 total, 8 up, 8 in 2024-03-12T03:22:49.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:22:49.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:49 smithi094 ceph-mon[40484]: osdmap e414: 8 total, 8 up, 8 in 2024-03-12T03:22:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:22:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:49 smithi151 ceph-mon[32923]: osdmap e413: 8 total, 8 up, 8 in 2024-03-12T03:22:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:22:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:49 smithi151 ceph-mon[32923]: osdmap e414: 8 total, 8 up, 8 in 2024-03-12T03:22:50.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:50 smithi094 ceph-mon[40484]: pgmap v600: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:50.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:50.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:50 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:50.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-67"}]: dispatch 2024-03-12T03:22:50.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:50 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-67"}]: dispatch 2024-03-12T03:22:50.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:50.531 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:50 smithi094 ceph-mon[29291]: pgmap v600: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:50.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:50.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:50 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:50.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-67"}]: dispatch 2024-03-12T03:22:50.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:50 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-67"}]: dispatch 2024-03-12T03:22:50.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:50 smithi151 ceph-mon[32923]: pgmap v600: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:50 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-67"}]: dispatch 2024-03-12T03:22:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:50 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-67"}]: dispatch 2024-03-12T03:22:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:50.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:22:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:22:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:22:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:51 smithi094 ceph-mon[40484]: osdmap e415: 8 total, 8 up, 8 in 2024-03-12T03:22:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:51 smithi094 ceph-mon[29291]: osdmap e415: 8 total, 8 up, 8 in 2024-03-12T03:22:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:51 smithi151 ceph-mon[32923]: osdmap e415: 8 total, 8 up, 8 in 2024-03-12T03:22:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:52 smithi094 ceph-mon[40484]: pgmap v603: 260 pgs: 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:22:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:52 smithi094 ceph-mon[40484]: osdmap e416: 8 total, 8 up, 8 in 2024-03-12T03:22:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:52 smithi094 ceph-mon[29291]: pgmap v603: 260 pgs: 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:22:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:52 smithi094 ceph-mon[29291]: osdmap e416: 8 total, 8 up, 8 in 2024-03-12T03:22:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:52 smithi151 ceph-mon[32923]: pgmap v603: 260 pgs: 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:22:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:52 smithi151 ceph-mon[32923]: osdmap e416: 8 total, 8 up, 8 in 2024-03-12T03:22:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:22:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[29291]: osdmap e417: 8 total, 8 up, 8 in 2024-03-12T03:22:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:22:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-69"}]: dispatch 2024-03-12T03:22:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-69"}]: dispatch 2024-03-12T03:22:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:22:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[40484]: osdmap e417: 8 total, 8 up, 8 in 2024-03-12T03:22:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:22:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-69"}]: dispatch 2024-03-12T03:22:53.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:53 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-69"}]: dispatch 2024-03-12T03:22:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:53 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:22:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:53 smithi151 ceph-mon[32923]: osdmap e417: 8 total, 8 up, 8 in 2024-03-12T03:22:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:22:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:53 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:22:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-69"}]: dispatch 2024-03-12T03:22:53.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:53 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-69"}]: dispatch 2024-03-12T03:22:54.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:54 smithi094 ceph-mon[29291]: pgmap v606: 292 pgs: 16 unknown, 276 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-12T03:22:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:54 smithi094 ceph-mon[29291]: osdmap e418: 8 total, 8 up, 8 in 2024-03-12T03:22:54.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:54 smithi094 ceph-mon[40484]: pgmap v606: 292 pgs: 16 unknown, 276 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-12T03:22:54.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:54 smithi094 ceph-mon[40484]: osdmap e418: 8 total, 8 up, 8 in 2024-03-12T03:22:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:54 smithi151 ceph-mon[32923]: pgmap v606: 292 pgs: 16 unknown, 276 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-12T03:22:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:54 smithi151 ceph-mon[32923]: osdmap e418: 8 total, 8 up, 8 in 2024-03-12T03:22:55.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:55 smithi094 ceph-mon[29291]: osdmap e419: 8 total, 8 up, 8 in 2024-03-12T03:22:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:55 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:55 smithi094 ceph-mon[40484]: osdmap e419: 8 total, 8 up, 8 in 2024-03-12T03:22:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:55 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:55 smithi151 ceph-mon[32923]: osdmap e419: 8 total, 8 up, 8 in 2024-03-12T03:22:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:55 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:22:56.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[40484]: pgmap v609: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:22:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:22:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[40484]: osdmap e420: 8 total, 8 up, 8 in 2024-03-12T03:22:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:22:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:22:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:22:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[29291]: pgmap v609: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:22:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:22:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[29291]: osdmap e420: 8 total, 8 up, 8 in 2024-03-12T03:22:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:22:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:22:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:56 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:22:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:56 smithi151 ceph-mon[32923]: pgmap v609: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:22:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:56 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:56 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:22:56.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:56 smithi151 ceph-mon[32923]: osdmap e420: 8 total, 8 up, 8 in 2024-03-12T03:22:56.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:22:56.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:22:56.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:56 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:22:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:57 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:22:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:57 smithi094 ceph-mon[29291]: osdmap e421: 8 total, 8 up, 8 in 2024-03-12T03:22:57.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:57 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:22:57.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:57 smithi094 ceph-mon[40484]: osdmap e421: 8 total, 8 up, 8 in 2024-03-12T03:22:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:57 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:22:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:57 smithi151 ceph-mon[32923]: osdmap e421: 8 total, 8 up, 8 in 2024-03-12T03:22:58.290 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:22:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:22:58.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:58 smithi094 ceph-mon[29291]: pgmap v612: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:58 smithi094 ceph-mon[29291]: osdmap e422: 8 total, 8 up, 8 in 2024-03-12T03:22:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:58 smithi094 ceph-mon[40484]: pgmap v612: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:58 smithi094 ceph-mon[40484]: osdmap e422: 8 total, 8 up, 8 in 2024-03-12T03:22:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:58 smithi151 ceph-mon[32923]: pgmap v612: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:22:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:58 smithi151 ceph-mon[32923]: osdmap e422: 8 total, 8 up, 8 in 2024-03-12T03:22:59.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:22:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:22:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:22:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:22:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:00.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:00 smithi094 ceph-mon[29291]: pgmap v614: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-12T03:23:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:00 smithi094 ceph-mon[40484]: pgmap v614: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-12T03:23:00.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:23:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:23:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:23:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:00 smithi151 ceph-mon[32923]: pgmap v614: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-12T03:23:02.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:02 smithi094 ceph-mon[29291]: pgmap v615: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-12T03:23:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:02 smithi094 ceph-mon[40484]: pgmap v615: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-12T03:23:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:02 smithi151 ceph-mon[32923]: pgmap v615: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-12T03:23:03.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:03 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:03 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:03 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:04.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:04 smithi094 ceph-mon[29291]: pgmap v616: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-12T03:23:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:04 smithi094 ceph-mon[40484]: pgmap v616: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-12T03:23:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:04 smithi151 ceph-mon[32923]: pgmap v616: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-12T03:23:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:06.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:06 smithi094 ceph-mon[29291]: pgmap v617: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-12T03:23:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:06 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:23:06.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:06 smithi094 ceph-mon[40484]: pgmap v617: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-12T03:23:06.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:06 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:23:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:06 smithi151 ceph-mon[32923]: pgmap v617: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-12T03:23:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:06 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:23:07.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:23:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:23:07.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:07 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:23:07.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:23:07.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:23:07.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:07 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:23:07.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:23:07.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:23:07.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:07 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:23:08.363 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:23:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:23:08.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:08 smithi094 ceph-mon[29291]: pgmap v618: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-12T03:23:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:08 smithi094 ceph-mon[40484]: pgmap v618: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-12T03:23:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:08 smithi151 ceph-mon[32923]: pgmap v618: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-12T03:23:09.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:09.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:10 smithi094 ceph-mon[29291]: pgmap v619: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-12T03:23:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:10 smithi094 ceph-mon[40484]: pgmap v619: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-12T03:23:10.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:23:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:23:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:23:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:10 smithi151 ceph-mon[32923]: pgmap v619: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-12T03:23:12.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:12 smithi094 ceph-mon[29291]: pgmap v620: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:23:12.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:12 smithi094 ceph-mon[40484]: pgmap v620: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:23:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:12 smithi151 ceph-mon[32923]: pgmap v620: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:23:14.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:14 smithi094 ceph-mon[40484]: pgmap v621: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:23:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:14 smithi094 ceph-mon[29291]: pgmap v621: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:23:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:14 smithi151 ceph-mon[32923]: pgmap v621: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:23:16.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:16 smithi094 ceph-mon[40484]: pgmap v622: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:23:16.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:16 smithi094 ceph-mon[29291]: pgmap v622: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:23:16.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:16 smithi151 ceph-mon[32923]: pgmap v622: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:23:17.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:17.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:17 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:17.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-71"}]: dispatch 2024-03-12T03:23:17.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:17 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-71"}]: dispatch 2024-03-12T03:23:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:17 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-71"}]: dispatch 2024-03-12T03:23:17.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:17 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-71"}]: dispatch 2024-03-12T03:23:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:17 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-71"}]: dispatch 2024-03-12T03:23:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:17 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-71"}]: dispatch 2024-03-12T03:23:18.386 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:23:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:23:18.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:18 smithi094 ceph-mon[29291]: pgmap v623: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:23:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:18 smithi094 ceph-mon[29291]: osdmap e423: 8 total, 8 up, 8 in 2024-03-12T03:23:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:18 smithi094 ceph-mon[40484]: pgmap v623: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:23:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:18 smithi094 ceph-mon[40484]: osdmap e423: 8 total, 8 up, 8 in 2024-03-12T03:23:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:18 smithi151 ceph-mon[32923]: pgmap v623: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:23:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:18 smithi151 ceph-mon[32923]: osdmap e423: 8 total, 8 up, 8 in 2024-03-12T03:23:19.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:19 smithi094 ceph-mon[29291]: osdmap e424: 8 total, 8 up, 8 in 2024-03-12T03:23:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:19 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:19 smithi094 ceph-mon[40484]: osdmap e424: 8 total, 8 up, 8 in 2024-03-12T03:23:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:19 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:19 smithi151 ceph-mon[32923]: osdmap e424: 8 total, 8 up, 8 in 2024-03-12T03:23:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:19 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:20.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:23:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:23:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:23:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:20 smithi094 ceph-mon[29291]: pgmap v626: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:23:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:20 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:20 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:23:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:20 smithi094 ceph-mon[29291]: osdmap e425: 8 total, 8 up, 8 in 2024-03-12T03:23:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:23:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:20 smithi094 ceph-mon[40484]: pgmap v626: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:23:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:20 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:20 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:23:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:20 smithi094 ceph-mon[40484]: osdmap e425: 8 total, 8 up, 8 in 2024-03-12T03:23:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:23:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:20 smithi151 ceph-mon[32923]: pgmap v626: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:23:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:20 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:20 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:23:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:20 smithi151 ceph-mon[32923]: osdmap e425: 8 total, 8 up, 8 in 2024-03-12T03:23:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:23:21.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:21 smithi094 ceph-mon[40484]: osdmap e426: 8 total, 8 up, 8 in 2024-03-12T03:23:21.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:21 smithi094 ceph-mon[29291]: osdmap e426: 8 total, 8 up, 8 in 2024-03-12T03:23:21.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:21 smithi151 ceph-mon[32923]: osdmap e426: 8 total, 8 up, 8 in 2024-03-12T03:23:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:22 smithi094 ceph-mon[29291]: pgmap v629: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:23:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:22 smithi094 ceph-mon[29291]: osdmap e427: 8 total, 8 up, 8 in 2024-03-12T03:23:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:22 smithi094 ceph-mon[40484]: pgmap v629: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:23:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:22 smithi094 ceph-mon[40484]: osdmap e427: 8 total, 8 up, 8 in 2024-03-12T03:23:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:22 smithi151 ceph-mon[32923]: pgmap v629: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:23:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:22 smithi151 ceph-mon[32923]: osdmap e427: 8 total, 8 up, 8 in 2024-03-12T03:23:24.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:24 smithi094 ceph-mon[29291]: pgmap v631: 292 pgs: 12 unknown, 280 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 715 B/s wr, 4 op/s 2024-03-12T03:23:24.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:24 smithi094 ceph-mon[40484]: pgmap v631: 292 pgs: 12 unknown, 280 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 715 B/s wr, 4 op/s 2024-03-12T03:23:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:24 smithi151 ceph-mon[32923]: pgmap v631: 292 pgs: 12 unknown, 280 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 715 B/s wr, 4 op/s 2024-03-12T03:23:26.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:26 smithi094 ceph-mon[29291]: pgmap v632: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-12T03:23:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:26 smithi094 ceph-mon[40484]: pgmap v632: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-12T03:23:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:26 smithi151 ceph-mon[32923]: pgmap v632: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-12T03:23:28.417 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:23:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:23:28.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:28 smithi094 ceph-mon[29291]: pgmap v633: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 983 B/s wr, 3 op/s 2024-03-12T03:23:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:28 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:28 smithi094 ceph-mon[40484]: pgmap v633: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 983 B/s wr, 3 op/s 2024-03-12T03:23:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:28 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:28 smithi151 ceph-mon[32923]: pgmap v633: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 983 B/s wr, 3 op/s 2024-03-12T03:23:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:28 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:30 smithi094 ceph-mon[29291]: pgmap v634: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 865 B/s wr, 3 op/s 2024-03-12T03:23:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:30 smithi094 ceph-mon[40484]: pgmap v634: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 865 B/s wr, 3 op/s 2024-03-12T03:23:30.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:23:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:23:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:23:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:30 smithi151 ceph-mon[32923]: pgmap v634: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 865 B/s wr, 3 op/s 2024-03-12T03:23:32.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:32 smithi094 ceph-mon[29291]: pgmap v635: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-03-12T03:23:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:32 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-73"}]: dispatch 2024-03-12T03:23:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:32 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-73"}]: dispatch 2024-03-12T03:23:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:32 smithi094 ceph-mon[40484]: pgmap v635: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-03-12T03:23:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:32 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-73"}]: dispatch 2024-03-12T03:23:32.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:32 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-73"}]: dispatch 2024-03-12T03:23:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:32 smithi151 ceph-mon[32923]: pgmap v635: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-03-12T03:23:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:32 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-73"}]: dispatch 2024-03-12T03:23:32.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:32 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-73"}]: dispatch 2024-03-12T03:23:33.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:33 smithi094 ceph-mon[29291]: osdmap e428: 8 total, 8 up, 8 in 2024-03-12T03:23:33.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:33 smithi094 ceph-mon[40484]: osdmap e428: 8 total, 8 up, 8 in 2024-03-12T03:23:33.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:33 smithi151 ceph-mon[32923]: osdmap e428: 8 total, 8 up, 8 in 2024-03-12T03:23:34.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:34 smithi094 ceph-mon[29291]: pgmap v637: 260 pgs: 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-12T03:23:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:34 smithi094 ceph-mon[29291]: osdmap e429: 8 total, 8 up, 8 in 2024-03-12T03:23:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:34 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:34 smithi094 ceph-mon[40484]: pgmap v637: 260 pgs: 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-12T03:23:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:34 smithi094 ceph-mon[40484]: osdmap e429: 8 total, 8 up, 8 in 2024-03-12T03:23:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:34 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:34 smithi151 ceph-mon[32923]: pgmap v637: 260 pgs: 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-12T03:23:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:34 smithi151 ceph-mon[32923]: osdmap e429: 8 total, 8 up, 8 in 2024-03-12T03:23:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:34 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:35.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:35 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:23:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:35 smithi094 ceph-mon[29291]: osdmap e430: 8 total, 8 up, 8 in 2024-03-12T03:23:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:23:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:35 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:23:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:35 smithi094 ceph-mon[40484]: osdmap e430: 8 total, 8 up, 8 in 2024-03-12T03:23:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:23:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:35 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:23:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:35 smithi151 ceph-mon[32923]: osdmap e430: 8 total, 8 up, 8 in 2024-03-12T03:23:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:23:36.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:36 smithi094 ceph-mon[29291]: pgmap v640: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:23:36.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:36 smithi094 ceph-mon[29291]: osdmap e431: 8 total, 8 up, 8 in 2024-03-12T03:23:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:36 smithi094 ceph-mon[40484]: pgmap v640: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:23:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:36 smithi094 ceph-mon[40484]: osdmap e431: 8 total, 8 up, 8 in 2024-03-12T03:23:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:36 smithi151 ceph-mon[32923]: pgmap v640: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:23:36.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:36 smithi151 ceph-mon[32923]: osdmap e431: 8 total, 8 up, 8 in 2024-03-12T03:23:37.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:37 smithi094 ceph-mon[40484]: osdmap e432: 8 total, 8 up, 8 in 2024-03-12T03:23:37.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:37.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:37 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:37.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-75"}]: dispatch 2024-03-12T03:23:37.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:37 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-75"}]: dispatch 2024-03-12T03:23:37.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:37 smithi094 ceph-mon[29291]: osdmap e432: 8 total, 8 up, 8 in 2024-03-12T03:23:37.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:37.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:37 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:37.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-75"}]: dispatch 2024-03-12T03:23:37.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:37 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-75"}]: dispatch 2024-03-12T03:23:37.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:37 smithi151 ceph-mon[32923]: osdmap e432: 8 total, 8 up, 8 in 2024-03-12T03:23:37.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:37.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:37 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:37.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-75"}]: dispatch 2024-03-12T03:23:37.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:37 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-75"}]: dispatch 2024-03-12T03:23:38.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:23:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:23:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:38 smithi094 ceph-mon[29291]: pgmap v643: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:23:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:38 smithi094 ceph-mon[29291]: osdmap e433: 8 total, 8 up, 8 in 2024-03-12T03:23:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:38 smithi094 ceph-mon[40484]: pgmap v643: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:23:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:38 smithi094 ceph-mon[40484]: osdmap e433: 8 total, 8 up, 8 in 2024-03-12T03:23:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:38 smithi151 ceph-mon[32923]: pgmap v643: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:23:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:38 smithi151 ceph-mon[32923]: osdmap e433: 8 total, 8 up, 8 in 2024-03-12T03:23:39.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:39.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:39 smithi094 ceph-mon[29291]: osdmap e434: 8 total, 8 up, 8 in 2024-03-12T03:23:39.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:39.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:39 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:39 smithi094 ceph-mon[40484]: osdmap e434: 8 total, 8 up, 8 in 2024-03-12T03:23:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:39 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:39 smithi151 ceph-mon[32923]: osdmap e434: 8 total, 8 up, 8 in 2024-03-12T03:23:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:39 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:40 smithi094 ceph-mon[29291]: pgmap v646: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:23:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:40 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:40 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:23:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:40 smithi094 ceph-mon[29291]: osdmap e435: 8 total, 8 up, 8 in 2024-03-12T03:23:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:23:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:23:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:23:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:23:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:40 smithi094 ceph-mon[40484]: pgmap v646: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:23:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:40 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:40 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:23:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:40 smithi094 ceph-mon[40484]: osdmap e435: 8 total, 8 up, 8 in 2024-03-12T03:23:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:23:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:40 smithi151 ceph-mon[32923]: pgmap v646: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:23:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:40 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:40 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:23:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:40 smithi151 ceph-mon[32923]: osdmap e435: 8 total, 8 up, 8 in 2024-03-12T03:23:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:23:41.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:41 smithi094 ceph-mon[29291]: osdmap e436: 8 total, 8 up, 8 in 2024-03-12T03:23:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:41 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-77"}]: dispatch 2024-03-12T03:23:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:41 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-77"}]: dispatch 2024-03-12T03:23:41.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:41 smithi094 ceph-mon[40484]: osdmap e436: 8 total, 8 up, 8 in 2024-03-12T03:23:41.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:41.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:41 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:41.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-77"}]: dispatch 2024-03-12T03:23:41.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:41 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-77"}]: dispatch 2024-03-12T03:23:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:41 smithi151 ceph-mon[32923]: osdmap e436: 8 total, 8 up, 8 in 2024-03-12T03:23:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:41 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:23:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-77"}]: dispatch 2024-03-12T03:23:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:41 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-77"}]: dispatch 2024-03-12T03:23:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:42 smithi094 ceph-mon[29291]: pgmap v649: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:23:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:42 smithi094 ceph-mon[29291]: osdmap e437: 8 total, 8 up, 8 in 2024-03-12T03:23:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:42 smithi094 ceph-mon[40484]: pgmap v649: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:23:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:42 smithi094 ceph-mon[40484]: osdmap e437: 8 total, 8 up, 8 in 2024-03-12T03:23:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:42 smithi151 ceph-mon[32923]: pgmap v649: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:23:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:42 smithi151 ceph-mon[32923]: osdmap e437: 8 total, 8 up, 8 in 2024-03-12T03:23:43.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:43 smithi094 ceph-mon[29291]: osdmap e438: 8 total, 8 up, 8 in 2024-03-12T03:23:43.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:43.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:43.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:23:43.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:43 smithi094 ceph-mon[29291]: osdmap e439: 8 total, 8 up, 8 in 2024-03-12T03:23:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:43 smithi094 ceph-mon[40484]: osdmap e438: 8 total, 8 up, 8 in 2024-03-12T03:23:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:43.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:23:43.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:43 smithi094 ceph-mon[40484]: osdmap e439: 8 total, 8 up, 8 in 2024-03-12T03:23:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:43 smithi151 ceph-mon[32923]: osdmap e438: 8 total, 8 up, 8 in 2024-03-12T03:23:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:23:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:23:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:43 smithi151 ceph-mon[32923]: osdmap e439: 8 total, 8 up, 8 in 2024-03-12T03:23:44.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:44 smithi151 ceph-mon[32923]: pgmap v652: 292 pgs: 18 creating+peering, 14 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:23:44.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:23:44.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:44 smithi151 ceph-mon[32923]: osdmap e440: 8 total, 8 up, 8 in 2024-03-12T03:23:44.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:44 smithi094 ceph-mon[29291]: pgmap v652: 292 pgs: 18 creating+peering, 14 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:23:44.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:23:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:44 smithi094 ceph-mon[29291]: osdmap e440: 8 total, 8 up, 8 in 2024-03-12T03:23:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:44 smithi094 ceph-mon[40484]: pgmap v652: 292 pgs: 18 creating+peering, 14 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:23:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:23:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:44 smithi094 ceph-mon[40484]: osdmap e440: 8 total, 8 up, 8 in 2024-03-12T03:23:46.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[29291]: pgmap v655: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:23:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[29291]: osdmap e441: 8 total, 8 up, 8 in 2024-03-12T03:23:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[29291]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "290.1"}]: dispatch 2024-03-12T03:23:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "290.1"}]: dispatch 2024-03-12T03:23:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[29291]: 290.1 deep-scrub starts 2024-03-12T03:23:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[29291]: 290.1 deep-scrub ok 2024-03-12T03:23:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[40484]: pgmap v655: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:23:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[40484]: osdmap e441: 8 total, 8 up, 8 in 2024-03-12T03:23:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[40484]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "290.1"}]: dispatch 2024-03-12T03:23:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "290.1"}]: dispatch 2024-03-12T03:23:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[40484]: 290.1 deep-scrub starts 2024-03-12T03:23:46.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:46 smithi094 ceph-mon[40484]: 290.1 deep-scrub ok 2024-03-12T03:23:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:46 smithi151 ceph-mon[32923]: pgmap v655: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:23:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:46 smithi151 ceph-mon[32923]: osdmap e441: 8 total, 8 up, 8 in 2024-03-12T03:23:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:46 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:23:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:46 smithi151 ceph-mon[32923]: from='mon.? v1:172.21.15.151:0/1995617038' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "290.1"}]: dispatch 2024-03-12T03:23:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "290.1"}]: dispatch 2024-03-12T03:23:46.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:46 smithi151 ceph-mon[32923]: 290.1 deep-scrub starts 2024-03-12T03:23:46.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:46 smithi151 ceph-mon[32923]: 290.1 deep-scrub ok 2024-03-12T03:23:48.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:23:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:23:48.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:48 smithi094 ceph-mon[29291]: pgmap v657: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 488 B/s wr, 2 op/s 2024-03-12T03:23:48.762 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:48 smithi094 ceph-mon[40484]: pgmap v657: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 488 B/s wr, 2 op/s 2024-03-12T03:23:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:48 smithi151 ceph-mon[32923]: pgmap v657: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 488 B/s wr, 2 op/s 2024-03-12T03:23:49.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:49.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:49 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:49.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:49.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:49 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:49.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:49.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:49 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:50 smithi094 ceph-mon[29291]: pgmap v658: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 1.2 KiB/s wr, 122 op/s 2024-03-12T03:23:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:50 smithi094 ceph-mon[40484]: pgmap v658: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 1.2 KiB/s wr, 122 op/s 2024-03-12T03:23:50.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:23:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:23:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:23:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:50 smithi151 ceph-mon[32923]: pgmap v658: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 1.2 KiB/s wr, 122 op/s 2024-03-12T03:23:52.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:52 smithi094 ceph-mon[29291]: pgmap v659: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 712 B/s wr, 101 op/s 2024-03-12T03:23:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:52 smithi094 ceph-mon[40484]: pgmap v659: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 712 B/s wr, 101 op/s 2024-03-12T03:23:52.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:52 smithi151 ceph-mon[32923]: pgmap v659: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 712 B/s wr, 101 op/s 2024-03-12T03:23:54.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:54 smithi094 ceph-mon[29291]: pgmap v660: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 625 B/s wr, 89 op/s 2024-03-12T03:23:54.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:54 smithi094 ceph-mon[40484]: pgmap v660: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 625 B/s wr, 89 op/s 2024-03-12T03:23:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:54 smithi151 ceph-mon[32923]: pgmap v660: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 625 B/s wr, 89 op/s 2024-03-12T03:23:56.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:56 smithi094 ceph-mon[29291]: pgmap v661: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 511 B/s wr, 73 op/s 2024-03-12T03:23:56.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:56 smithi094 ceph-mon[40484]: pgmap v661: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 511 B/s wr, 73 op/s 2024-03-12T03:23:56.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:56 smithi151 ceph-mon[32923]: pgmap v661: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 511 B/s wr, 73 op/s 2024-03-12T03:23:58.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:23:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:23:58.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:58 smithi094 ceph-mon[29291]: pgmap v662: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 457 B/s wr, 65 op/s 2024-03-12T03:23:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:58 smithi094 ceph-mon[40484]: pgmap v662: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 457 B/s wr, 65 op/s 2024-03-12T03:23:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:58 smithi151 ceph-mon[32923]: pgmap v662: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 457 B/s wr, 65 op/s 2024-03-12T03:23:59.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:23:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:59.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:23:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:23:59.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:23:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:00 smithi094 ceph-mon[29291]: pgmap v663: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 426 B/s wr, 61 op/s 2024-03-12T03:24:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:00 smithi094 ceph-mon[40484]: pgmap v663: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 426 B/s wr, 61 op/s 2024-03-12T03:24:00.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:24:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:24:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:24:00.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:00 smithi151 ceph-mon[32923]: pgmap v663: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 426 B/s wr, 61 op/s 2024-03-12T03:24:02.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:02 smithi094 ceph-mon[29291]: pgmap v664: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:24:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:02 smithi094 ceph-mon[40484]: pgmap v664: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:24:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:02 smithi151 ceph-mon[32923]: pgmap v664: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:24:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:04 smithi094 ceph-mon[29291]: pgmap v665: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:24:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:04 smithi094 ceph-mon[40484]: pgmap v665: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:24:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:04 smithi151 ceph-mon[32923]: pgmap v665: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:24:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:05.960 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:24:05 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3[62481]: 2024-03-12T03:24:05.541+0000 7f6842289700 -1 snap_mapper.add_oid found existing snaps mapped on 290:83207d4c:test-rados-api-smithi094-78452-80::foo:2, removing 2024-03-12T03:24:05.964 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:24:05 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:24:05.541+0000 7f2703723700 -1 snap_mapper.add_oid found existing snaps mapped on 290:83207d4c:test-rados-api-smithi094-78452-80::foo:2, removing 2024-03-12T03:24:05.964 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:24:05 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:24:05.542+0000 7f74902d3700 -1 snap_mapper.add_oid found existing snaps mapped on 290:83207d4c:test-rados-api-smithi094-78452-80::foo:2, removing 2024-03-12T03:24:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:06 smithi094 ceph-mon[29291]: pgmap v666: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:24:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:06 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-79"}]: dispatch 2024-03-12T03:24:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:06 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-79"}]: dispatch 2024-03-12T03:24:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:06 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:24:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:06 smithi094 ceph-mon[40484]: pgmap v666: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:24:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:06 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-79"}]: dispatch 2024-03-12T03:24:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:06 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-79"}]: dispatch 2024-03-12T03:24:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:06 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:24:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:06 smithi151 ceph-mon[32923]: pgmap v666: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:24:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:06 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-79"}]: dispatch 2024-03-12T03:24:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:06 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-79"}]: dispatch 2024-03-12T03:24:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:06 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:24:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:07 smithi094 ceph-mon[29291]: osdmap e442: 8 total, 8 up, 8 in 2024-03-12T03:24:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:24:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:24:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:07 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:24:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:07 smithi094 ceph-mon[40484]: osdmap e442: 8 total, 8 up, 8 in 2024-03-12T03:24:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:24:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:24:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:07 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:24:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:07 smithi151 ceph-mon[32923]: osdmap e442: 8 total, 8 up, 8 in 2024-03-12T03:24:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:24:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:24:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:07 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:24:08.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:24:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:24:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:08 smithi094 ceph-mon[29291]: pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:24:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:08 smithi094 ceph-mon[29291]: osdmap e443: 8 total, 8 up, 8 in 2024-03-12T03:24:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:08 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:08 smithi094 ceph-mon[40484]: pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:24:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:08 smithi094 ceph-mon[40484]: osdmap e443: 8 total, 8 up, 8 in 2024-03-12T03:24:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:08 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:08 smithi151 ceph-mon[32923]: pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:24:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:08 smithi151 ceph-mon[32923]: osdmap e443: 8 total, 8 up, 8 in 2024-03-12T03:24:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:08 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:09.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:09 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:09 smithi094 ceph-mon[29291]: osdmap e444: 8 total, 8 up, 8 in 2024-03-12T03:24:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:09 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:09 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:09 smithi094 ceph-mon[40484]: osdmap e444: 8 total, 8 up, 8 in 2024-03-12T03:24:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:09.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:09 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:09 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:09 smithi151 ceph-mon[32923]: osdmap e444: 8 total, 8 up, 8 in 2024-03-12T03:24:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:09 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:24:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:24:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:24:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:10 smithi094 ceph-mon[29291]: pgmap v671: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T03:24:10.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:10 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:10.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:10.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:10 smithi094 ceph-mon[29291]: osdmap e445: 8 total, 8 up, 8 in 2024-03-12T03:24:10.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:10 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:10.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:10 smithi094 ceph-mon[40484]: pgmap v671: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T03:24:10.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:10 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:10.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:10.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:10 smithi094 ceph-mon[40484]: osdmap e445: 8 total, 8 up, 8 in 2024-03-12T03:24:10.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:10 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:10 smithi151 ceph-mon[32923]: pgmap v671: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T03:24:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:10 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:10 smithi151 ceph-mon[32923]: osdmap e445: 8 total, 8 up, 8 in 2024-03-12T03:24:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:10 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:11 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:24:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:11 smithi094 ceph-mon[29291]: osdmap e446: 8 total, 8 up, 8 in 2024-03-12T03:24:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:11 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:11 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:24:11.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:11 smithi094 ceph-mon[29291]: osdmap e447: 8 total, 8 up, 8 in 2024-03-12T03:24:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:11 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:24:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:11 smithi094 ceph-mon[40484]: osdmap e446: 8 total, 8 up, 8 in 2024-03-12T03:24:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:11 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:11 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:24:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:11 smithi094 ceph-mon[40484]: osdmap e447: 8 total, 8 up, 8 in 2024-03-12T03:24:11.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:11 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:24:11.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:11 smithi151 ceph-mon[32923]: osdmap e446: 8 total, 8 up, 8 in 2024-03-12T03:24:11.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:11.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:11 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:11.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:11 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:24:11.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:11 smithi151 ceph-mon[32923]: osdmap e447: 8 total, 8 up, 8 in 2024-03-12T03:24:12.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:12 smithi094 ceph-mon[29291]: pgmap v674: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-12T03:24:12.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:12.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:12 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:12.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:12 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:12.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:12 smithi094 ceph-mon[29291]: osdmap e448: 8 total, 8 up, 8 in 2024-03-12T03:24:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:12 smithi094 ceph-mon[40484]: pgmap v674: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-12T03:24:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:12 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:12 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:12.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:12 smithi094 ceph-mon[40484]: osdmap e448: 8 total, 8 up, 8 in 2024-03-12T03:24:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:12 smithi151 ceph-mon[32923]: pgmap v674: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-12T03:24:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:12 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:12 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:12.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:12 smithi151 ceph-mon[32923]: osdmap e448: 8 total, 8 up, 8 in 2024-03-12T03:24:13.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-12T03:24:13.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:13 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-12T03:24:13.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-12T03:24:13.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:13 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-12T03:24:13.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-12T03:24:13.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:13 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-12T03:24:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:14 smithi151 ceph-mon[32923]: pgmap v677: 292 pgs: 12 creating+peering, 280 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 4 op/s 2024-03-12T03:24:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:14 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-12T03:24:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:14 smithi151 ceph-mon[32923]: osdmap e449: 8 total, 8 up, 8 in 2024-03-12T03:24:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-12T03:24:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:14 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-12T03:24:14.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:14 smithi094 ceph-mon[29291]: pgmap v677: 292 pgs: 12 creating+peering, 280 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 4 op/s 2024-03-12T03:24:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:14 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-12T03:24:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:14 smithi094 ceph-mon[29291]: osdmap e449: 8 total, 8 up, 8 in 2024-03-12T03:24:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-12T03:24:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:14 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-12T03:24:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:14 smithi094 ceph-mon[40484]: pgmap v677: 292 pgs: 12 creating+peering, 280 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 4 op/s 2024-03-12T03:24:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:14 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-12T03:24:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:14 smithi094 ceph-mon[40484]: osdmap e449: 8 total, 8 up, 8 in 2024-03-12T03:24:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-12T03:24:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:14 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-12T03:24:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:15 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-12T03:24:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:15 smithi094 ceph-mon[29291]: osdmap e450: 8 total, 8 up, 8 in 2024-03-12T03:24:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:15 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:15 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:15 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-12T03:24:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:15 smithi094 ceph-mon[40484]: osdmap e450: 8 total, 8 up, 8 in 2024-03-12T03:24:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:15 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:15 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:15 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-12T03:24:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:15 smithi151 ceph-mon[32923]: osdmap e450: 8 total, 8 up, 8 in 2024-03-12T03:24:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:15 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:15 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:16.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[40484]: pgmap v680: 292 pgs: 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-12T03:24:16.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:16.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[40484]: osdmap e451: 8 total, 8 up, 8 in 2024-03-12T03:24:16.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:16.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:16.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-81"}]: dispatch 2024-03-12T03:24:16.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-81"}]: dispatch 2024-03-12T03:24:16.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[29291]: pgmap v680: 292 pgs: 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-12T03:24:16.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:16.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[29291]: osdmap e451: 8 total, 8 up, 8 in 2024-03-12T03:24:16.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:16.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:16.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-81"}]: dispatch 2024-03-12T03:24:16.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:16 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-81"}]: dispatch 2024-03-12T03:24:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:16 smithi151 ceph-mon[32923]: pgmap v680: 292 pgs: 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-12T03:24:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:16 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:16 smithi151 ceph-mon[32923]: osdmap e451: 8 total, 8 up, 8 in 2024-03-12T03:24:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:16 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-81"}]: dispatch 2024-03-12T03:24:16.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:16 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-81"}]: dispatch 2024-03-12T03:24:17.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:17 smithi094 ceph-mon[40484]: osdmap e452: 8 total, 8 up, 8 in 2024-03-12T03:24:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:17 smithi094 ceph-mon[29291]: osdmap e452: 8 total, 8 up, 8 in 2024-03-12T03:24:17.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:17 smithi151 ceph-mon[32923]: osdmap e452: 8 total, 8 up, 8 in 2024-03-12T03:24:18.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:24:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:24:18.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:18 smithi094 ceph-mon[29291]: pgmap v683: 260 pgs: 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 3 op/s 2024-03-12T03:24:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:18 smithi094 ceph-mon[29291]: osdmap e453: 8 total, 8 up, 8 in 2024-03-12T03:24:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:18 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:18 smithi094 ceph-mon[40484]: pgmap v683: 260 pgs: 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 3 op/s 2024-03-12T03:24:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:18 smithi094 ceph-mon[40484]: osdmap e453: 8 total, 8 up, 8 in 2024-03-12T03:24:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:18 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:18 smithi151 ceph-mon[32923]: pgmap v683: 260 pgs: 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 3 op/s 2024-03-12T03:24:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:18 smithi151 ceph-mon[32923]: osdmap e453: 8 total, 8 up, 8 in 2024-03-12T03:24:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:18 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[29291]: osdmap e454: 8 total, 8 up, 8 in 2024-03-12T03:24:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:19.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:19.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[40484]: osdmap e454: 8 total, 8 up, 8 in 2024-03-12T03:24:19.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:19.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:19.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:19.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:19 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:19 smithi151 ceph-mon[32923]: osdmap e454: 8 total, 8 up, 8 in 2024-03-12T03:24:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:19.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:19 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:19.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:19 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:19.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:20.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:20 smithi094 ceph-mon[40484]: pgmap v686: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-12T03:24:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:20 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:20 smithi094 ceph-mon[40484]: osdmap e455: 8 total, 8 up, 8 in 2024-03-12T03:24:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:20 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:20 smithi094 ceph-mon[29291]: pgmap v686: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-12T03:24:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:20 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:20.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:20.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:20 smithi094 ceph-mon[29291]: osdmap e455: 8 total, 8 up, 8 in 2024-03-12T03:24:20.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:20 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:20.961 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:24:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:24:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:24:20.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:20 smithi151 ceph-mon[32923]: pgmap v686: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-12T03:24:20.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:20 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:20.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:20.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:20 smithi151 ceph-mon[32923]: osdmap e455: 8 total, 8 up, 8 in 2024-03-12T03:24:20.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:20 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:21 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:24:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:21 smithi094 ceph-mon[29291]: osdmap e456: 8 total, 8 up, 8 in 2024-03-12T03:24:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:21 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:21 smithi094 ceph-mon[29291]: pgmap v689: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-03-12T03:24:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:21 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:24:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:21 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:24:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:21 smithi094 ceph-mon[40484]: osdmap e456: 8 total, 8 up, 8 in 2024-03-12T03:24:21.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:21.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:21 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:21.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:21 smithi094 ceph-mon[40484]: pgmap v689: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-03-12T03:24:21.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:21 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:24:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:21 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:24:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:21 smithi151 ceph-mon[32923]: osdmap e456: 8 total, 8 up, 8 in 2024-03-12T03:24:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:21 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:21 smithi151 ceph-mon[32923]: pgmap v689: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-03-12T03:24:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:21 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:24:22.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:22 smithi094 ceph-mon[29291]: osdmap e457: 8 total, 8 up, 8 in 2024-03-12T03:24:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:22 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:22 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:22 smithi094 ceph-mon[29291]: osdmap e458: 8 total, 8 up, 8 in 2024-03-12T03:24:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:22 smithi094 ceph-mon[40484]: osdmap e457: 8 total, 8 up, 8 in 2024-03-12T03:24:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:22 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:22 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:22 smithi094 ceph-mon[40484]: osdmap e458: 8 total, 8 up, 8 in 2024-03-12T03:24:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:22 smithi151 ceph-mon[32923]: osdmap e457: 8 total, 8 up, 8 in 2024-03-12T03:24:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:22 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:22 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:22 smithi151 ceph-mon[32923]: osdmap e458: 8 total, 8 up, 8 in 2024-03-12T03:24:23.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:23 smithi094 ceph-mon[29291]: pgmap v692: 292 pgs: 17 creating+peering, 275 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-12T03:24:23.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:23 smithi094 ceph-mon[40484]: pgmap v692: 292 pgs: 17 creating+peering, 275 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-12T03:24:23.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:23 smithi151 ceph-mon[32923]: pgmap v692: 292 pgs: 17 creating+peering, 275 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-12T03:24:24.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:24 smithi094 ceph-mon[29291]: osdmap e459: 8 total, 8 up, 8 in 2024-03-12T03:24:24.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:24 smithi094 ceph-mon[40484]: osdmap e459: 8 total, 8 up, 8 in 2024-03-12T03:24:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:24 smithi151 ceph-mon[32923]: osdmap e459: 8 total, 8 up, 8 in 2024-03-12T03:24:25.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:25 smithi094 ceph-mon[29291]: osdmap e460: 8 total, 8 up, 8 in 2024-03-12T03:24:25.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:25.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:25 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:25.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-83"}]: dispatch 2024-03-12T03:24:25.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:25 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-83"}]: dispatch 2024-03-12T03:24:25.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:25 smithi094 ceph-mon[29291]: pgmap v695: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-12T03:24:25.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:25 smithi094 ceph-mon[40484]: osdmap e460: 8 total, 8 up, 8 in 2024-03-12T03:24:25.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:25.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:25 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:25.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-83"}]: dispatch 2024-03-12T03:24:25.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:25 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-83"}]: dispatch 2024-03-12T03:24:25.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:25 smithi094 ceph-mon[40484]: pgmap v695: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-12T03:24:25.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:25 smithi151 ceph-mon[32923]: osdmap e460: 8 total, 8 up, 8 in 2024-03-12T03:24:25.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:25.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:25 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:25.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-83"}]: dispatch 2024-03-12T03:24:25.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:25 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-83"}]: dispatch 2024-03-12T03:24:25.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:25 smithi151 ceph-mon[32923]: pgmap v695: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-12T03:24:26.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:26 smithi094 ceph-mon[29291]: osdmap e461: 8 total, 8 up, 8 in 2024-03-12T03:24:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:26 smithi094 ceph-mon[40484]: osdmap e461: 8 total, 8 up, 8 in 2024-03-12T03:24:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:26 smithi151 ceph-mon[32923]: osdmap e461: 8 total, 8 up, 8 in 2024-03-12T03:24:27.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:27 smithi094 ceph-mon[40484]: osdmap e462: 8 total, 8 up, 8 in 2024-03-12T03:24:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:27 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:27 smithi094 ceph-mon[40484]: pgmap v698: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:24:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:27 smithi094 ceph-mon[29291]: osdmap e462: 8 total, 8 up, 8 in 2024-03-12T03:24:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:27 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:27 smithi094 ceph-mon[29291]: pgmap v698: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:24:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:27 smithi151 ceph-mon[32923]: osdmap e462: 8 total, 8 up, 8 in 2024-03-12T03:24:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:27 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:27 smithi151 ceph-mon[32923]: pgmap v698: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:24:28.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:24:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:24:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[29291]: osdmap e463: 8 total, 8 up, 8 in 2024-03-12T03:24:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[40484]: osdmap e463: 8 total, 8 up, 8 in 2024-03-12T03:24:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:28.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:28.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:28.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:28 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:28 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:28.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:28 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:28.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:28 smithi151 ceph-mon[32923]: osdmap e463: 8 total, 8 up, 8 in 2024-03-12T03:24:28.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:28.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:28.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:28 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:28.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:28 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:29.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:29 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:29 smithi094 ceph-mon[29291]: osdmap e464: 8 total, 8 up, 8 in 2024-03-12T03:24:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:29 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:29 smithi094 ceph-mon[29291]: pgmap v701: 292 pgs: 16 creating+peering, 276 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-12T03:24:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:29 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:29 smithi094 ceph-mon[40484]: osdmap e464: 8 total, 8 up, 8 in 2024-03-12T03:24:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:29.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:29 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:29.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:29 smithi094 ceph-mon[40484]: pgmap v701: 292 pgs: 16 creating+peering, 276 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-12T03:24:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:29 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:29 smithi151 ceph-mon[32923]: osdmap e464: 8 total, 8 up, 8 in 2024-03-12T03:24:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:29 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:29 smithi151 ceph-mon[32923]: pgmap v701: 292 pgs: 16 creating+peering, 276 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-12T03:24:30.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:24:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:24:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:24:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:30 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:24:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:30 smithi094 ceph-mon[29291]: osdmap e465: 8 total, 8 up, 8 in 2024-03-12T03:24:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:30 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:30 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:24:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:30 smithi094 ceph-mon[40484]: osdmap e465: 8 total, 8 up, 8 in 2024-03-12T03:24:30.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:30 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:30.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:30 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_tier","val": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:24:30.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:30.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:30 smithi151 ceph-mon[32923]: osdmap e465: 8 total, 8 up, 8 in 2024-03-12T03:24:30.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:30 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:31.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:24:31.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[29291]: osdmap e466: 8 total, 8 up, 8 in 2024-03-12T03:24:31.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:31.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:31.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[29291]: pgmap v704: 292 pgs: 16 creating+peering, 276 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-12T03:24:31.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:31.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[29291]: osdmap e467: 8 total, 8 up, 8 in 2024-03-12T03:24:31.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:24:31.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[40484]: osdmap e466: 8 total, 8 up, 8 in 2024-03-12T03:24:31.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:31.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:31.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[40484]: pgmap v704: 292 pgs: 16 creating+peering, 276 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-12T03:24:31.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:31.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:31 smithi094 ceph-mon[40484]: osdmap e467: 8 total, 8 up, 8 in 2024-03-12T03:24:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:31 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:24:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:31 smithi151 ceph-mon[32923]: osdmap e466: 8 total, 8 up, 8 in 2024-03-12T03:24:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:31 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:31 smithi151 ceph-mon[32923]: pgmap v704: 292 pgs: 16 creating+peering, 276 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-12T03:24:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:31 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:31.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:31 smithi151 ceph-mon[32923]: osdmap e467: 8 total, 8 up, 8 in 2024-03-12T03:24:33.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:33 smithi094 ceph-mon[29291]: osdmap e468: 8 total, 8 up, 8 in 2024-03-12T03:24:33.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:33 smithi094 ceph-mon[29291]: pgmap v707: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-12T03:24:33.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:33 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:33 smithi094 ceph-mon[40484]: osdmap e468: 8 total, 8 up, 8 in 2024-03-12T03:24:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:33 smithi094 ceph-mon[40484]: pgmap v707: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-12T03:24:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:33 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:33.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:33 smithi151 ceph-mon[32923]: osdmap e468: 8 total, 8 up, 8 in 2024-03-12T03:24:33.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:33 smithi151 ceph-mon[32923]: pgmap v707: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-12T03:24:33.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:33 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:34 smithi151 ceph-mon[32923]: osdmap e469: 8 total, 8 up, 8 in 2024-03-12T03:24:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:34 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:34.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-85"}]: dispatch 2024-03-12T03:24:34.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:34 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-85"}]: dispatch 2024-03-12T03:24:34.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:35.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:34 smithi094 ceph-mon[40484]: osdmap e469: 8 total, 8 up, 8 in 2024-03-12T03:24:35.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:35.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:34 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:35.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-85"}]: dispatch 2024-03-12T03:24:35.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:34 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-85"}]: dispatch 2024-03-12T03:24:35.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:34 smithi094 ceph-mon[29291]: osdmap e469: 8 total, 8 up, 8 in 2024-03-12T03:24:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:34 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-85"}]: dispatch 2024-03-12T03:24:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:34 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-85"}]: dispatch 2024-03-12T03:24:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:36.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:35 smithi094 ceph-mon[29291]: pgmap v709: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-12T03:24:36.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:35 smithi094 ceph-mon[29291]: osdmap e470: 8 total, 8 up, 8 in 2024-03-12T03:24:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:35 smithi094 ceph-mon[40484]: pgmap v709: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-12T03:24:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:35 smithi094 ceph-mon[40484]: osdmap e470: 8 total, 8 up, 8 in 2024-03-12T03:24:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:35 smithi151 ceph-mon[32923]: pgmap v709: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-12T03:24:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:35 smithi151 ceph-mon[32923]: osdmap e470: 8 total, 8 up, 8 in 2024-03-12T03:24:36.740 INFO:tasks.workunit.client.0.smithi094.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18346 ms) 2024-03-12T03:24:36.740 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-12T03:24:36.740 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (21949 ms) 2024-03-12T03:24:36.740 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-12T03:24:36.740 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3017 ms) 2024-03-12T03:24:36.740 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-12T03:24:36.740 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.740 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3025 ms) 2024-03-12T03:24:36.740 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-12T03:24:36.740 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3018 ms) 2024-03-12T03:24:36.740 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-12T03:24:36.740 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.741 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3020 ms) 2024-03-12T03:24:36.741 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-12T03:24:36.741 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.741 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4037 ms) 2024-03-12T03:24:36.741 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-12T03:24:36.741 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.741 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38350 ms) 2024-03-12T03:24:36.741 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-12T03:24:36.741 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.741 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17006 ms) 2024-03-12T03:24:36.741 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-12T03:24:36.741 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4044 ms) 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3024 ms) 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24123 ms) 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15043 ms) 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5037 ms) 2024-03-12T03:24:36.742 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4026 ms) 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: waiting for scrubs... 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: done waiting 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25045 ms) 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10062 ms) 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9060 ms) 2024-03-12T03:24:36.743 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-12T03:24:36.744 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:24:36.744 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9050 ms) 2024-03-12T03:24:36.744 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-12T03:24:37.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:36 smithi094 ceph-mon[40484]: osdmap e471: 8 total, 8 up, 8 in 2024-03-12T03:24:37.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:36 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:37.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:36 smithi094 ceph-mon[29291]: osdmap e471: 8 total, 8 up, 8 in 2024-03-12T03:24:37.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:37.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:36 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:36 smithi151 ceph-mon[32923]: osdmap e471: 8 total, 8 up, 8 in 2024-03-12T03:24:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:36 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:38.126 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:37 smithi151 ceph-mon[32923]: pgmap v712: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-03-12T03:24:38.126 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:37 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:38.127 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:37 smithi151 ceph-mon[32923]: osdmap e472: 8 total, 8 up, 8 in 2024-03-12T03:24:38.127 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:38.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:37 smithi094 ceph-mon[29291]: pgmap v712: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-03-12T03:24:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:37 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:37 smithi094 ceph-mon[29291]: osdmap e472: 8 total, 8 up, 8 in 2024-03-12T03:24:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:37 smithi094 ceph-mon[40484]: pgmap v712: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-03-12T03:24:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:37 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:37 smithi094 ceph-mon[40484]: osdmap e472: 8 total, 8 up, 8 in 2024-03-12T03:24:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:38.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:24:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:24:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[40484]: osdmap e473: 8 total, 8 up, 8 in 2024-03-12T03:24:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_tier","val": "test-rados-api-smithi094-78452-89-test-flush"}]: dispatch 2024-03-12T03:24:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[40484]: osdmap e474: 8 total, 8 up, 8 in 2024-03-12T03:24:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_tier","val": "test-rados-api-smithi094-78452-89-test-flush"}]: dispatch 2024-03-12T03:24:39.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[29291]: osdmap e473: 8 total, 8 up, 8 in 2024-03-12T03:24:39.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:39.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:39.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:39.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:39.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:39.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_tier","val": "test-rados-api-smithi094-78452-89-test-flush"}]: dispatch 2024-03-12T03:24:39.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[29291]: osdmap e474: 8 total, 8 up, 8 in 2024-03-12T03:24:39.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:38 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_tier","val": "test-rados-api-smithi094-78452-89-test-flush"}]: dispatch 2024-03-12T03:24:39.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:38 smithi151 ceph-mon[32923]: osdmap e473: 8 total, 8 up, 8 in 2024-03-12T03:24:39.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:39.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:38 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:39.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:38 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:39.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:38 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:39.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:38 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:39.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_tier","val": "test-rados-api-smithi094-78452-89-test-flush"}]: dispatch 2024-03-12T03:24:39.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:38 smithi151 ceph-mon[32923]: osdmap e474: 8 total, 8 up, 8 in 2024-03-12T03:24:39.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:38 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_tier","val": "test-rados-api-smithi094-78452-89-test-flush"}]: dispatch 2024-03-12T03:24:40.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:39 smithi094 ceph-mon[29291]: pgmap v715: 324 pgs: 20 creating+peering, 12 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-03-12T03:24:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:39 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_tier","val": "test-rados-api-smithi094-78452-89-test-flush"}]': finished 2024-03-12T03:24:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:39 smithi094 ceph-mon[29291]: osdmap e475: 8 total, 8 up, 8 in 2024-03-12T03:24:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:39 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:40.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:39 smithi094 ceph-mon[40484]: pgmap v715: 324 pgs: 20 creating+peering, 12 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-03-12T03:24:40.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:39 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_tier","val": "test-rados-api-smithi094-78452-89-test-flush"}]': finished 2024-03-12T03:24:40.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:39 smithi094 ceph-mon[40484]: osdmap e475: 8 total, 8 up, 8 in 2024-03-12T03:24:40.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:40.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:39 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:40.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:39 smithi151 ceph-mon[32923]: pgmap v715: 324 pgs: 20 creating+peering, 12 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-03-12T03:24:40.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:39 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_tier","val": "test-rados-api-smithi094-78452-89-test-flush"}]': finished 2024-03-12T03:24:40.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:39 smithi151 ceph-mon[32923]: osdmap e475: 8 total, 8 up, 8 in 2024-03-12T03:24:40.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:40.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:39 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:24:40.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:24:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:24:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:24:42.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:41 smithi094 ceph-mon[40484]: pgmap v718: 324 pgs: 20 creating+peering, 12 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-12T03:24:42.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:41 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:24:42.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:41 smithi094 ceph-mon[40484]: osdmap e476: 8 total, 8 up, 8 in 2024-03-12T03:24:42.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:42.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:41 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:41 smithi094 ceph-mon[29291]: pgmap v718: 324 pgs: 20 creating+peering, 12 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-12T03:24:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:41 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:24:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:41 smithi094 ceph-mon[29291]: osdmap e476: 8 total, 8 up, 8 in 2024-03-12T03:24:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:41 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:41 smithi151 ceph-mon[32923]: pgmap v718: 324 pgs: 20 creating+peering, 12 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-12T03:24:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:41 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:24:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:41 smithi151 ceph-mon[32923]: osdmap e476: 8 total, 8 up, 8 in 2024-03-12T03:24:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:41 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:24:43.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:42 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:42 smithi094 ceph-mon[29291]: osdmap e477: 8 total, 8 up, 8 in 2024-03-12T03:24:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:42 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:42 smithi094 ceph-mon[40484]: osdmap e477: 8 total, 8 up, 8 in 2024-03-12T03:24:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:42 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:24:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:42 smithi151 ceph-mon[32923]: osdmap e477: 8 total, 8 up, 8 in 2024-03-12T03:24:44.115 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:43 smithi151 ceph-mon[32923]: pgmap v721: 324 pgs: 20 creating+peering, 304 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:24:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:43 smithi151 ceph-mon[32923]: osdmap e478: 8 total, 8 up, 8 in 2024-03-12T03:24:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-87"}]: dispatch 2024-03-12T03:24:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-87"}]: dispatch 2024-03-12T03:24:44.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:43 smithi094 ceph-mon[40484]: pgmap v721: 324 pgs: 20 creating+peering, 304 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:24:44.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:43 smithi094 ceph-mon[40484]: osdmap e478: 8 total, 8 up, 8 in 2024-03-12T03:24:44.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:44.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:44.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-87"}]: dispatch 2024-03-12T03:24:44.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-87"}]: dispatch 2024-03-12T03:24:44.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:43 smithi094 ceph-mon[29291]: pgmap v721: 324 pgs: 20 creating+peering, 304 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:24:44.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:43 smithi094 ceph-mon[29291]: osdmap e478: 8 total, 8 up, 8 in 2024-03-12T03:24:44.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:44.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:44.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-87"}]: dispatch 2024-03-12T03:24:44.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-87"}]: dispatch 2024-03-12T03:24:45.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:44 smithi094 ceph-mon[40484]: osdmap e479: 8 total, 8 up, 8 in 2024-03-12T03:24:45.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:44 smithi094 ceph-mon[29291]: osdmap e479: 8 total, 8 up, 8 in 2024-03-12T03:24:45.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:44 smithi151 ceph-mon[32923]: osdmap e479: 8 total, 8 up, 8 in 2024-03-12T03:24:46.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:45 smithi094 ceph-mon[40484]: pgmap v724: 260 pgs: 260 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:24:46.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:45 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:46.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:45 smithi094 ceph-mon[40484]: osdmap e480: 8 total, 8 up, 8 in 2024-03-12T03:24:46.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:46.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:45 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:45 smithi094 ceph-mon[29291]: pgmap v724: 260 pgs: 260 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:24:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:45 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:45 smithi094 ceph-mon[29291]: osdmap e480: 8 total, 8 up, 8 in 2024-03-12T03:24:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:46.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:45 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:46.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:45 smithi151 ceph-mon[32923]: pgmap v724: 260 pgs: 260 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:24:46.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:45 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:46.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:45 smithi151 ceph-mon[32923]: osdmap e480: 8 total, 8 up, 8 in 2024-03-12T03:24:46.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:46.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:45 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:47.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:46 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:46 smithi094 ceph-mon[29291]: osdmap e481: 8 total, 8 up, 8 in 2024-03-12T03:24:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:46 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:46 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:46 smithi094 ceph-mon[40484]: osdmap e481: 8 total, 8 up, 8 in 2024-03-12T03:24:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:46 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:46 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:46 smithi151 ceph-mon[32923]: osdmap e481: 8 total, 8 up, 8 in 2024-03-12T03:24:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:46 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:24:48.111 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:47 smithi094 ceph-mon[40484]: pgmap v727: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-03-12T03:24:48.111 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:47 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:48.111 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:47 smithi094 ceph-mon[40484]: osdmap e482: 8 total, 8 up, 8 in 2024-03-12T03:24:48.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:47 smithi094 ceph-mon[29291]: pgmap v727: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-03-12T03:24:48.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:47 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:48.112 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:47 smithi094 ceph-mon[29291]: osdmap e482: 8 total, 8 up, 8 in 2024-03-12T03:24:48.134 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:47 smithi151 ceph-mon[32923]: pgmap v727: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-03-12T03:24:48.134 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:47 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:24:48.135 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:47 smithi151 ceph-mon[32923]: osdmap e482: 8 total, 8 up, 8 in 2024-03-12T03:24:48.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:24:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:24:49.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:48 smithi094 ceph-mon[29291]: osdmap e483: 8 total, 8 up, 8 in 2024-03-12T03:24:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:48 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:48 smithi094 ceph-mon[40484]: osdmap e483: 8 total, 8 up, 8 in 2024-03-12T03:24:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:48 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:48 smithi151 ceph-mon[32923]: osdmap e483: 8 total, 8 up, 8 in 2024-03-12T03:24:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:48 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:50.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[40484]: pgmap v730: 292 pgs: 292 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-12T03:24:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[40484]: osdmap e484: 8 total, 8 up, 8 in 2024-03-12T03:24:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-90"}]: dispatch 2024-03-12T03:24:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-90"}]: dispatch 2024-03-12T03:24:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:50.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[29291]: pgmap v730: 292 pgs: 292 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-12T03:24:50.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[29291]: osdmap e484: 8 total, 8 up, 8 in 2024-03-12T03:24:50.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:50.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:50.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-90"}]: dispatch 2024-03-12T03:24:50.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-90"}]: dispatch 2024-03-12T03:24:50.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:49 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:49 smithi151 ceph-mon[32923]: pgmap v730: 292 pgs: 292 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-12T03:24:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:49 smithi151 ceph-mon[32923]: osdmap e484: 8 total, 8 up, 8 in 2024-03-12T03:24:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-90"}]: dispatch 2024-03-12T03:24:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-90"}]: dispatch 2024-03-12T03:24:50.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:49 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:50.795 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:24:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:24:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:24:51.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:50 smithi094 ceph-mon[29291]: osdmap e485: 8 total, 8 up, 8 in 2024-03-12T03:24:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:50 smithi094 ceph-mon[40484]: osdmap e485: 8 total, 8 up, 8 in 2024-03-12T03:24:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:50 smithi151 ceph-mon[32923]: osdmap e485: 8 total, 8 up, 8 in 2024-03-12T03:24:52.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:51 smithi094 ceph-mon[29291]: pgmap v733: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:24:52.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:51 smithi094 ceph-mon[29291]: osdmap e486: 8 total, 8 up, 8 in 2024-03-12T03:24:52.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:52.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:51 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:52.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:51 smithi094 ceph-mon[40484]: pgmap v733: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:24:52.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:51 smithi094 ceph-mon[40484]: osdmap e486: 8 total, 8 up, 8 in 2024-03-12T03:24:52.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:52.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:51 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:52.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:51 smithi151 ceph-mon[32923]: pgmap v733: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:24:52.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:51 smithi151 ceph-mon[32923]: osdmap e486: 8 total, 8 up, 8 in 2024-03-12T03:24:52.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:52.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:51 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:53.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:53.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:52 smithi094 ceph-mon[29291]: osdmap e487: 8 total, 8 up, 8 in 2024-03-12T03:24:53.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:53.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:53.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:52 smithi094 ceph-mon[40484]: osdmap e487: 8 total, 8 up, 8 in 2024-03-12T03:24:53.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:53.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:53.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:52 smithi151 ceph-mon[32923]: osdmap e487: 8 total, 8 up, 8 in 2024-03-12T03:24:53.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:54.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:53 smithi094 ceph-mon[29291]: pgmap v736: 292 pgs: 12 unknown, 280 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:24:54.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:53 smithi094 ceph-mon[29291]: osdmap e488: 8 total, 8 up, 8 in 2024-03-12T03:24:54.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:53 smithi094 ceph-mon[40484]: pgmap v736: 292 pgs: 12 unknown, 280 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:24:54.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:53 smithi094 ceph-mon[40484]: osdmap e488: 8 total, 8 up, 8 in 2024-03-12T03:24:54.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:53 smithi151 ceph-mon[32923]: pgmap v736: 292 pgs: 12 unknown, 280 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:24:54.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:53 smithi151 ceph-mon[32923]: osdmap e488: 8 total, 8 up, 8 in 2024-03-12T03:24:55.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:54 smithi094 ceph-mon[29291]: osdmap e489: 8 total, 8 up, 8 in 2024-03-12T03:24:55.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:55.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:54 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:55.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-92"}]: dispatch 2024-03-12T03:24:55.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:54 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-92"}]: dispatch 2024-03-12T03:24:55.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:54 smithi094 ceph-mon[40484]: osdmap e489: 8 total, 8 up, 8 in 2024-03-12T03:24:55.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:55.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:54 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:55.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-92"}]: dispatch 2024-03-12T03:24:55.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:54 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-92"}]: dispatch 2024-03-12T03:24:55.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:54 smithi151 ceph-mon[32923]: osdmap e489: 8 total, 8 up, 8 in 2024-03-12T03:24:55.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:55.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:54 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:24:55.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-92"}]: dispatch 2024-03-12T03:24:55.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:54 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-92"}]: dispatch 2024-03-12T03:24:56.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:55 smithi094 ceph-mon[29291]: pgmap v739: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-12T03:24:56.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:55 smithi094 ceph-mon[29291]: osdmap e490: 8 total, 8 up, 8 in 2024-03-12T03:24:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:55 smithi094 ceph-mon[40484]: pgmap v739: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-12T03:24:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:55 smithi094 ceph-mon[40484]: osdmap e490: 8 total, 8 up, 8 in 2024-03-12T03:24:56.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:55 smithi151 ceph-mon[32923]: pgmap v739: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-12T03:24:56.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:55 smithi151 ceph-mon[32923]: osdmap e490: 8 total, 8 up, 8 in 2024-03-12T03:24:57.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:56 smithi094 ceph-mon[29291]: osdmap e491: 8 total, 8 up, 8 in 2024-03-12T03:24:57.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:57.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:56 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:57.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:56 smithi094 ceph-mon[40484]: osdmap e491: 8 total, 8 up, 8 in 2024-03-12T03:24:57.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:57.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:56 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:57.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:56 smithi151 ceph-mon[32923]: osdmap e491: 8 total, 8 up, 8 in 2024-03-12T03:24:57.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:57.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:56 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:24:58.145 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:57 smithi151 ceph-mon[32923]: pgmap v742: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:24:58.145 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:57 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:58.145 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:57 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:58.145 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:57 smithi151 ceph-mon[32923]: osdmap e492: 8 total, 8 up, 8 in 2024-03-12T03:24:58.146 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:58.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:57 smithi094 ceph-mon[40484]: pgmap v742: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:24:58.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:57 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:58.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:57 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:58.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:57 smithi094 ceph-mon[40484]: osdmap e492: 8 total, 8 up, 8 in 2024-03-12T03:24:58.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:58.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:57 smithi094 ceph-mon[29291]: pgmap v742: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:24:58.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:57 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:24:58.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:57 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:24:58.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:57 smithi094 ceph-mon[29291]: osdmap e492: 8 total, 8 up, 8 in 2024-03-12T03:24:58.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:24:58.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:24:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:24:59.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:58 smithi094 ceph-mon[29291]: osdmap e493: 8 total, 8 up, 8 in 2024-03-12T03:24:59.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:58 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:59.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:58 smithi094 ceph-mon[40484]: osdmap e493: 8 total, 8 up, 8 in 2024-03-12T03:24:59.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:58 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:24:59.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:58 smithi151 ceph-mon[32923]: osdmap e493: 8 total, 8 up, 8 in 2024-03-12T03:24:59.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:58 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:00.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:59 smithi094 ceph-mon[40484]: pgmap v745: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:25:00.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:24:59 smithi094 ceph-mon[40484]: osdmap e494: 8 total, 8 up, 8 in 2024-03-12T03:25:00.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:59 smithi094 ceph-mon[29291]: pgmap v745: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:25:00.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:24:59 smithi094 ceph-mon[29291]: osdmap e494: 8 total, 8 up, 8 in 2024-03-12T03:25:00.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:59 smithi151 ceph-mon[32923]: pgmap v745: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:25:00.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:24:59 smithi151 ceph-mon[32923]: osdmap e494: 8 total, 8 up, 8 in 2024-03-12T03:25:00.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:25:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:25:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:25:02.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:01 smithi094 ceph-mon[40484]: pgmap v747: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-03-12T03:25:02.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:01 smithi094 ceph-mon[29291]: pgmap v747: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-03-12T03:25:02.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:01 smithi151 ceph-mon[32923]: pgmap v747: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-03-12T03:25:04.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:03 smithi094 ceph-mon[40484]: pgmap v748: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-12T03:25:04.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:03 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:25:04.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:03 smithi094 ceph-mon[29291]: pgmap v748: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-12T03:25:04.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:03 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:25:04.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:03 smithi151 ceph-mon[32923]: pgmap v748: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-12T03:25:04.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:03 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:25:05.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:25:05.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:25:05.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:25:06.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:05 smithi094 ceph-mon[29291]: pgmap v749: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-12T03:25:06.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:05 smithi094 ceph-mon[40484]: pgmap v749: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-12T03:25:06.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:05 smithi151 ceph-mon[32923]: pgmap v749: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-12T03:25:08.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: pgmap v750: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 230 B/s wr, 2 op/s 2024-03-12T03:25:08.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:25:08.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.155 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:25:08.156 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:25:08.157 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:07 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:25:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: pgmap v750: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 230 B/s wr, 2 op/s 2024-03-12T03:25:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:25:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:25:08.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:25:08.212 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:25:08.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: pgmap v750: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 230 B/s wr, 2 op/s 2024-03-12T03:25:08.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:25:08.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.212 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:25:08.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:25:08.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:25:08.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:25:08.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:25:08.213 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:07 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:25:08.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:25:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:25:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:09 smithi094 ceph-mon[29291]: Adjusting osd_memory_target on smithi094 to 3248M 2024-03-12T03:25:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:09 smithi094 ceph-mon[29291]: Adjusting osd_memory_target on smithi151 to 3248M 2024-03-12T03:25:09.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:09.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:09 smithi094 ceph-mon[40484]: Adjusting osd_memory_target on smithi094 to 3248M 2024-03-12T03:25:09.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:09 smithi094 ceph-mon[40484]: Adjusting osd_memory_target on smithi151 to 3248M 2024-03-12T03:25:09.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:09 smithi151 ceph-mon[32923]: Adjusting osd_memory_target on smithi094 to 3248M 2024-03-12T03:25:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:09 smithi151 ceph-mon[32923]: Adjusting osd_memory_target on smithi151 to 3248M 2024-03-12T03:25:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:10.531 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:10 smithi094 ceph-mon[29291]: pgmap v751: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-12T03:25:10.531 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:10 smithi094 ceph-mon[29291]: osdmap e495: 8 total, 8 up, 8 in 2024-03-12T03:25:10.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:10 smithi094 ceph-mon[40484]: pgmap v751: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-12T03:25:10.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:10 smithi094 ceph-mon[40484]: osdmap e495: 8 total, 8 up, 8 in 2024-03-12T03:25:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:10 smithi151 ceph-mon[32923]: pgmap v751: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-12T03:25:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:10 smithi151 ceph-mon[32923]: osdmap e495: 8 total, 8 up, 8 in 2024-03-12T03:25:10.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:25:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:25:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:25:12.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:12 smithi094 ceph-mon[40484]: pgmap v753: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-12T03:25:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:12 smithi094 ceph-mon[29291]: pgmap v753: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-12T03:25:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:12 smithi151 ceph-mon[32923]: pgmap v753: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-12T03:25:14.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:14 smithi151 ceph-mon[32923]: pgmap v754: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-12T03:25:14.465 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:25:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=cleanup t=2024-03-12T03:25:14.016610739Z level=info msg="Completed cleanup jobs" duration=3.499972ms 2024-03-12T03:25:14.465 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:25:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=sqlstore.transactions t=2024-03-12T03:25:14.023798649Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-12T03:25:14.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:14 smithi094 ceph-mon[29291]: pgmap v754: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-12T03:25:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:14 smithi094 ceph-mon[40484]: pgmap v754: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-12T03:25:16.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:16 smithi094 ceph-mon[29291]: pgmap v755: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:25:16.835 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:16 smithi094 ceph-mon[40484]: pgmap v755: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:25:16.836 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:16 smithi151 ceph-mon[32923]: pgmap v755: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:25:18.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:18 smithi151 ceph-mon[32923]: pgmap v756: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:25:18.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:18 smithi151 ceph-mon[32923]: osdmap e496: 8 total, 8 up, 8 in 2024-03-12T03:25:18.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:25:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:25:18.512 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:18 smithi094 ceph-mon[40484]: pgmap v756: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:25:18.512 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:18 smithi094 ceph-mon[40484]: osdmap e496: 8 total, 8 up, 8 in 2024-03-12T03:25:18.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:18 smithi094 ceph-mon[29291]: pgmap v756: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:25:18.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:18 smithi094 ceph-mon[29291]: osdmap e496: 8 total, 8 up, 8 in 2024-03-12T03:25:19.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:20.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:20 smithi094 ceph-mon[40484]: pgmap v758: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:25:20.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:20.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:20 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:20.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-94"}]: dispatch 2024-03-12T03:25:20.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:20 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-94"}]: dispatch 2024-03-12T03:25:20.531 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:25:20.531 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:20 smithi094 ceph-mon[29291]: pgmap v758: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:25:20.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:20.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:20 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:20.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-94"}]: dispatch 2024-03-12T03:25:20.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:20 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-94"}]: dispatch 2024-03-12T03:25:20.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:25:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:20 smithi151 ceph-mon[32923]: pgmap v758: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:25:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:20 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-94"}]: dispatch 2024-03-12T03:25:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:20 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-94"}]: dispatch 2024-03-12T03:25:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:25:20.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:25:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:25:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:25:21.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:21 smithi094 ceph-mon[29291]: osdmap e497: 8 total, 8 up, 8 in 2024-03-12T03:25:21.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:21 smithi094 ceph-mon[40484]: osdmap e497: 8 total, 8 up, 8 in 2024-03-12T03:25:21.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:21 smithi151 ceph-mon[32923]: osdmap e497: 8 total, 8 up, 8 in 2024-03-12T03:25:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:22 smithi094 ceph-mon[29291]: pgmap v760: 260 pgs: 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:25:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:22 smithi094 ceph-mon[29291]: osdmap e498: 8 total, 8 up, 8 in 2024-03-12T03:25:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:22 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:22.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:22 smithi094 ceph-mon[40484]: pgmap v760: 260 pgs: 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:25:22.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:22 smithi094 ceph-mon[40484]: osdmap e498: 8 total, 8 up, 8 in 2024-03-12T03:25:22.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:22.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:22 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:22 smithi151 ceph-mon[32923]: pgmap v760: 260 pgs: 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:25:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:22 smithi151 ceph-mon[32923]: osdmap e498: 8 total, 8 up, 8 in 2024-03-12T03:25:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:22 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:23.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:23 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:25:23.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:23 smithi094 ceph-mon[29291]: osdmap e499: 8 total, 8 up, 8 in 2024-03-12T03:25:23.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:25:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:23 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:25:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:23 smithi094 ceph-mon[40484]: osdmap e499: 8 total, 8 up, 8 in 2024-03-12T03:25:23.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:25:23.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:23 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:25:23.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:23 smithi151 ceph-mon[32923]: osdmap e499: 8 total, 8 up, 8 in 2024-03-12T03:25:23.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:25:24.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:24 smithi094 ceph-mon[29291]: pgmap v763: 292 pgs: 15 unknown, 277 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:25:24.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:24 smithi094 ceph-mon[29291]: osdmap e500: 8 total, 8 up, 8 in 2024-03-12T03:25:24.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:24 smithi094 ceph-mon[40484]: pgmap v763: 292 pgs: 15 unknown, 277 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:25:24.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:24 smithi094 ceph-mon[40484]: osdmap e500: 8 total, 8 up, 8 in 2024-03-12T03:25:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:24 smithi151 ceph-mon[32923]: pgmap v763: 292 pgs: 15 unknown, 277 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:25:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:24 smithi151 ceph-mon[32923]: osdmap e500: 8 total, 8 up, 8 in 2024-03-12T03:25:26.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:26 smithi094 ceph-mon[29291]: pgmap v765: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-12T03:25:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:26 smithi094 ceph-mon[40484]: pgmap v765: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-12T03:25:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:26 smithi151 ceph-mon[32923]: pgmap v765: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-12T03:25:28.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:28 smithi151 ceph-mon[32923]: pgmap v766: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 1 op/s 2024-03-12T03:25:28.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:25:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:25:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:28 smithi094 ceph-mon[29291]: pgmap v766: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 1 op/s 2024-03-12T03:25:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:28 smithi094 ceph-mon[40484]: pgmap v766: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 1 op/s 2024-03-12T03:25:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:30.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:30 smithi094 ceph-mon[29291]: pgmap v767: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 823 B/s wr, 2 op/s 2024-03-12T03:25:30.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:25:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:25:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:25:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:30 smithi094 ceph-mon[40484]: pgmap v767: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 823 B/s wr, 2 op/s 2024-03-12T03:25:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:30 smithi151 ceph-mon[32923]: pgmap v767: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 823 B/s wr, 2 op/s 2024-03-12T03:25:32.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:32 smithi094 ceph-mon[29291]: pgmap v768: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 726 B/s wr, 2 op/s 2024-03-12T03:25:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:32 smithi094 ceph-mon[40484]: pgmap v768: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 726 B/s wr, 2 op/s 2024-03-12T03:25:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:32 smithi151 ceph-mon[32923]: pgmap v768: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 726 B/s wr, 2 op/s 2024-03-12T03:25:33.710 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:25:33 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1[50221]: 2024-03-12T03:25:33.309+0000 7f2b28ebc700 -1 snap_mapper.add_oid found existing snaps mapped on 106:413804f8:test-rados-api-smithi094-78452-97::foo:21, removing 2024-03-12T03:25:33.710 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:25:33 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3[62481]: 2024-03-12T03:25:33.309+0000 7f6841a88700 -1 snap_mapper.add_oid found existing snaps mapped on 106:413804f8:test-rados-api-smithi094-78452-97::foo:21, removing 2024-03-12T03:25:33.714 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:25:33 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:25:33.310+0000 7f74932d9700 -1 snap_mapper.add_oid found existing snaps mapped on 106:413804f8:test-rados-api-smithi094-78452-97::foo:21, removing 2024-03-12T03:25:34.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:34 smithi094 ceph-mon[29291]: pgmap v769: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 409 B/s wr, 1 op/s 2024-03-12T03:25:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:34 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-96"}]: dispatch 2024-03-12T03:25:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:34 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-96"}]: dispatch 2024-03-12T03:25:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:34 smithi094 ceph-mon[40484]: pgmap v769: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 409 B/s wr, 1 op/s 2024-03-12T03:25:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:34 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-96"}]: dispatch 2024-03-12T03:25:34.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:34 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-96"}]: dispatch 2024-03-12T03:25:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:34 smithi151 ceph-mon[32923]: pgmap v769: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 409 B/s wr, 1 op/s 2024-03-12T03:25:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:34 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-96"}]: dispatch 2024-03-12T03:25:34.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:34 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-96"}]: dispatch 2024-03-12T03:25:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:35 smithi094 ceph-mon[40484]: osdmap e501: 8 total, 8 up, 8 in 2024-03-12T03:25:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:25:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:35 smithi094 ceph-mon[29291]: osdmap e501: 8 total, 8 up, 8 in 2024-03-12T03:25:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:25:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:35 smithi151 ceph-mon[32923]: osdmap e501: 8 total, 8 up, 8 in 2024-03-12T03:25:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:25:36.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:36 smithi094 ceph-mon[29291]: pgmap v771: 260 pgs: 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:25:36.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:36 smithi094 ceph-mon[29291]: osdmap e502: 8 total, 8 up, 8 in 2024-03-12T03:25:36.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:36.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:36 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:36 smithi094 ceph-mon[40484]: pgmap v771: 260 pgs: 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:25:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:36 smithi094 ceph-mon[40484]: osdmap e502: 8 total, 8 up, 8 in 2024-03-12T03:25:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:36 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:36 smithi151 ceph-mon[32923]: pgmap v771: 260 pgs: 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:25:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:36 smithi151 ceph-mon[32923]: osdmap e502: 8 total, 8 up, 8 in 2024-03-12T03:25:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:36 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:37.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:37 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:25:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:37 smithi094 ceph-mon[29291]: osdmap e503: 8 total, 8 up, 8 in 2024-03-12T03:25:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:25:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:37 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:25:37.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:37 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:25:37.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:37 smithi094 ceph-mon[40484]: osdmap e503: 8 total, 8 up, 8 in 2024-03-12T03:25:37.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:25:37.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:37 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:25:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:37 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:25:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:37 smithi151 ceph-mon[32923]: osdmap e503: 8 total, 8 up, 8 in 2024-03-12T03:25:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:25:37.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:37 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:25:38.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:38 smithi151 ceph-mon[32923]: pgmap v774: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:25:38.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:38 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:25:38.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:38 smithi151 ceph-mon[32923]: osdmap e504: 8 total, 8 up, 8 in 2024-03-12T03:25:38.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-98", "mode": "writeback"}]: dispatch 2024-03-12T03:25:38.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:38 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-98", "mode": "writeback"}]: dispatch 2024-03-12T03:25:38.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:25:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:25:38.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:38 smithi094 ceph-mon[40484]: pgmap v774: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:25:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:38 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:25:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:38 smithi094 ceph-mon[40484]: osdmap e504: 8 total, 8 up, 8 in 2024-03-12T03:25:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-98", "mode": "writeback"}]: dispatch 2024-03-12T03:25:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:38 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-98", "mode": "writeback"}]: dispatch 2024-03-12T03:25:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:38 smithi094 ceph-mon[29291]: pgmap v774: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:25:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:38 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:25:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:38 smithi094 ceph-mon[29291]: osdmap e504: 8 total, 8 up, 8 in 2024-03-12T03:25:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-98", "mode": "writeback"}]: dispatch 2024-03-12T03:25:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:38 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-98", "mode": "writeback"}]: dispatch 2024-03-12T03:25:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:39 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:25:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:39 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-98", "mode": "writeback"}]': finished 2024-03-12T03:25:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:39 smithi094 ceph-mon[40484]: osdmap e505: 8 total, 8 up, 8 in 2024-03-12T03:25:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-98"}]: dispatch 2024-03-12T03:25:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:39 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-98"}]: dispatch 2024-03-12T03:25:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:39 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:25:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:39 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-98", "mode": "writeback"}]': finished 2024-03-12T03:25:39.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:39 smithi094 ceph-mon[29291]: osdmap e505: 8 total, 8 up, 8 in 2024-03-12T03:25:39.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-98"}]: dispatch 2024-03-12T03:25:39.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:39 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-98"}]: dispatch 2024-03-12T03:25:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:39 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:25:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:39 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-98", "mode": "writeback"}]': finished 2024-03-12T03:25:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:39 smithi151 ceph-mon[32923]: osdmap e505: 8 total, 8 up, 8 in 2024-03-12T03:25:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-98"}]: dispatch 2024-03-12T03:25:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:39 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-98"}]: dispatch 2024-03-12T03:25:40.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:25:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:25:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:25:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:40 smithi094 ceph-mon[29291]: pgmap v777: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:25:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:40 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-98"}]': finished 2024-03-12T03:25:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:25:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:40 smithi094 ceph-mon[29291]: osdmap e506: 8 total, 8 up, 8 in 2024-03-12T03:25:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:40 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:25:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:40 smithi094 ceph-mon[40484]: pgmap v777: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:25:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:40 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-98"}]': finished 2024-03-12T03:25:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:25:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:40 smithi094 ceph-mon[40484]: osdmap e506: 8 total, 8 up, 8 in 2024-03-12T03:25:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:40 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:25:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:40 smithi151 ceph-mon[32923]: pgmap v777: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:25:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:40 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-98"}]': finished 2024-03-12T03:25:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:25:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:40 smithi151 ceph-mon[32923]: osdmap e506: 8 total, 8 up, 8 in 2024-03-12T03:25:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:40 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:25:41.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:41 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:25:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:41 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:25:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:41 smithi094 ceph-mon[29291]: osdmap e507: 8 total, 8 up, 8 in 2024-03-12T03:25:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-12T03:25:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:41 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-12T03:25:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:41 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:25:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:41 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:25:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:41 smithi094 ceph-mon[40484]: osdmap e507: 8 total, 8 up, 8 in 2024-03-12T03:25:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-12T03:25:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:41 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-12T03:25:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:41 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:25:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:41 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:25:41.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:41 smithi151 ceph-mon[32923]: osdmap e507: 8 total, 8 up, 8 in 2024-03-12T03:25:41.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-12T03:25:41.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:41 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-12T03:25:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:42 smithi094 ceph-mon[29291]: pgmap v780: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:25:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:42 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_count","val": "1"}]': finished 2024-03-12T03:25:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:42 smithi094 ceph-mon[29291]: osdmap e508: 8 total, 8 up, 8 in 2024-03-12T03:25:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:25:42.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:42 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:25:42.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:42 smithi094 ceph-mon[40484]: pgmap v780: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:25:42.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:42 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_count","val": "1"}]': finished 2024-03-12T03:25:42.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:42 smithi094 ceph-mon[40484]: osdmap e508: 8 total, 8 up, 8 in 2024-03-12T03:25:42.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:25:42.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:42 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:25:42.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:42 smithi151 ceph-mon[32923]: pgmap v780: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:25:42.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:42 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_count","val": "1"}]': finished 2024-03-12T03:25:42.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:42 smithi151 ceph-mon[32923]: osdmap e508: 8 total, 8 up, 8 in 2024-03-12T03:25:42.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:25:42.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:42 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:25:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:25:43.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:43 smithi094 ceph-mon[29291]: osdmap e509: 8 total, 8 up, 8 in 2024-03-12T03:25:43.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-12T03:25:43.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-12T03:25:43.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:43 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "target_max_objects","val": "250"}]': finished 2024-03-12T03:25:43.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:43 smithi094 ceph-mon[29291]: osdmap e510: 8 total, 8 up, 8 in 2024-03-12T03:25:43.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:25:43.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:43 smithi094 ceph-mon[40484]: osdmap e509: 8 total, 8 up, 8 in 2024-03-12T03:25:43.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-12T03:25:43.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-12T03:25:43.712 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:43 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "target_max_objects","val": "250"}]': finished 2024-03-12T03:25:43.713 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:43 smithi094 ceph-mon[40484]: osdmap e510: 8 total, 8 up, 8 in 2024-03-12T03:25:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:25:43.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:43 smithi151 ceph-mon[32923]: osdmap e509: 8 total, 8 up, 8 in 2024-03-12T03:25:43.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-12T03:25:43.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-12T03:25:43.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:43 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-98","var": "target_max_objects","val": "250"}]': finished 2024-03-12T03:25:43.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:43 smithi151 ceph-mon[32923]: osdmap e510: 8 total, 8 up, 8 in 2024-03-12T03:25:44.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:44 smithi094 ceph-mon[29291]: pgmap v783: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:44 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:44 smithi094 ceph-mon[40484]: pgmap v783: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:44 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:44 smithi151 ceph-mon[32923]: pgmap v783: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:44 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:45.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:45 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:25:45.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:45 smithi094 ceph-mon[40484]: osdmap e511: 8 total, 8 up, 8 in 2024-03-12T03:25:45.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98"}]: dispatch 2024-03-12T03:25:45.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:45 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98"}]: dispatch 2024-03-12T03:25:45.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:45 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:25:45.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:45 smithi094 ceph-mon[29291]: osdmap e511: 8 total, 8 up, 8 in 2024-03-12T03:25:45.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98"}]: dispatch 2024-03-12T03:25:45.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:45 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98"}]: dispatch 2024-03-12T03:25:45.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:45 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:25:45.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:45 smithi151 ceph-mon[32923]: osdmap e511: 8 total, 8 up, 8 in 2024-03-12T03:25:45.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98"}]: dispatch 2024-03-12T03:25:45.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:45 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98"}]: dispatch 2024-03-12T03:25:46.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:46 smithi094 ceph-mon[29291]: pgmap v786: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:46.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:46 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98"}]': finished 2024-03-12T03:25:46.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:46 smithi094 ceph-mon[29291]: osdmap e512: 8 total, 8 up, 8 in 2024-03-12T03:25:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:46 smithi094 ceph-mon[40484]: pgmap v786: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:46 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98"}]': finished 2024-03-12T03:25:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:46 smithi094 ceph-mon[40484]: osdmap e512: 8 total, 8 up, 8 in 2024-03-12T03:25:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:46 smithi151 ceph-mon[32923]: pgmap v786: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:46 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-98"}]': finished 2024-03-12T03:25:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:46 smithi151 ceph-mon[32923]: osdmap e512: 8 total, 8 up, 8 in 2024-03-12T03:25:47.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:47 smithi094 ceph-mon[29291]: osdmap e513: 8 total, 8 up, 8 in 2024-03-12T03:25:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:47 smithi094 ceph-mon[40484]: osdmap e513: 8 total, 8 up, 8 in 2024-03-12T03:25:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:47 smithi151 ceph-mon[32923]: osdmap e513: 8 total, 8 up, 8 in 2024-03-12T03:25:48.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:25:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:25:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:48 smithi151 ceph-mon[32923]: pgmap v789: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:25:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:48 smithi151 ceph-mon[32923]: osdmap e514: 8 total, 8 up, 8 in 2024-03-12T03:25:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:48.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:48 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:48.612 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:48 smithi094 ceph-mon[29291]: pgmap v789: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:25:48.612 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:48 smithi094 ceph-mon[29291]: osdmap e514: 8 total, 8 up, 8 in 2024-03-12T03:25:48.612 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:48.612 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:48 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:48.612 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:48 smithi094 ceph-mon[40484]: pgmap v789: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:25:48.613 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:48 smithi094 ceph-mon[40484]: osdmap e514: 8 total, 8 up, 8 in 2024-03-12T03:25:48.613 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:48.613 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:48 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:25:49.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:25:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:49 smithi094 ceph-mon[40484]: osdmap e515: 8 total, 8 up, 8 in 2024-03-12T03:25:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:25:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:49 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:25:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:25:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:49 smithi094 ceph-mon[29291]: osdmap e515: 8 total, 8 up, 8 in 2024-03-12T03:25:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:25:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:49 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:25:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:25:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:49 smithi151 ceph-mon[32923]: osdmap e515: 8 total, 8 up, 8 in 2024-03-12T03:25:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:25:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:49 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:25:50.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:25:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:25:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:25:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:50 smithi094 ceph-mon[40484]: pgmap v792: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:25:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:50 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:25:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-100"}]: dispatch 2024-03-12T03:25:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:50 smithi094 ceph-mon[40484]: osdmap e516: 8 total, 8 up, 8 in 2024-03-12T03:25:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:50 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-100"}]: dispatch 2024-03-12T03:25:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:50 smithi094 ceph-mon[29291]: pgmap v792: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:25:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:50 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:25:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-100"}]: dispatch 2024-03-12T03:25:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:50 smithi094 ceph-mon[29291]: osdmap e516: 8 total, 8 up, 8 in 2024-03-12T03:25:50.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:50 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-100"}]: dispatch 2024-03-12T03:25:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:50 smithi151 ceph-mon[32923]: pgmap v792: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:25:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:50 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:25:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-100"}]: dispatch 2024-03-12T03:25:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:50 smithi151 ceph-mon[32923]: osdmap e516: 8 total, 8 up, 8 in 2024-03-12T03:25:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:50 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-100"}]: dispatch 2024-03-12T03:25:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-100"}]': finished 2024-03-12T03:25:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[29291]: osdmap e517: 8 total, 8 up, 8 in 2024-03-12T03:25:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-100", "mode": "writeback"}]: dispatch 2024-03-12T03:25:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-100", "mode": "writeback"}]: dispatch 2024-03-12T03:25:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:25:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-100", "mode": "writeback"}]': finished 2024-03-12T03:25:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[29291]: osdmap e518: 8 total, 8 up, 8 in 2024-03-12T03:25:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-100"}]': finished 2024-03-12T03:25:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[40484]: osdmap e517: 8 total, 8 up, 8 in 2024-03-12T03:25:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-100", "mode": "writeback"}]: dispatch 2024-03-12T03:25:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-100", "mode": "writeback"}]: dispatch 2024-03-12T03:25:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:25:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-100", "mode": "writeback"}]': finished 2024-03-12T03:25:51.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:51 smithi094 ceph-mon[40484]: osdmap e518: 8 total, 8 up, 8 in 2024-03-12T03:25:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:51 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-6", "overlaypool": "test-rados-api-smithi094-78452-100"}]': finished 2024-03-12T03:25:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:51 smithi151 ceph-mon[32923]: osdmap e517: 8 total, 8 up, 8 in 2024-03-12T03:25:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-100", "mode": "writeback"}]: dispatch 2024-03-12T03:25:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:51 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-100", "mode": "writeback"}]: dispatch 2024-03-12T03:25:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:51 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:25:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:51 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-100", "mode": "writeback"}]': finished 2024-03-12T03:25:51.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:51 smithi151 ceph-mon[32923]: osdmap e518: 8 total, 8 up, 8 in 2024-03-12T03:25:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[29291]: pgmap v795: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:25:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:25:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:25:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[29291]: osdmap e519: 8 total, 8 up, 8 in 2024-03-12T03:25:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:25:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:25:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[40484]: pgmap v795: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:25:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:25:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:25:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[40484]: osdmap e519: 8 total, 8 up, 8 in 2024-03-12T03:25:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:25:52.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:52 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:25:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:52 smithi151 ceph-mon[32923]: pgmap v795: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:25:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:25:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:25:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:52 smithi151 ceph-mon[32923]: osdmap e519: 8 total, 8 up, 8 in 2024-03-12T03:25:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:25:52.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:52 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:25:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:54 smithi094 ceph-mon[29291]: pgmap v798: 292 pgs: 14 creating+peering, 278 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:54 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:25:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:54 smithi094 ceph-mon[29291]: osdmap e520: 8 total, 8 up, 8 in 2024-03-12T03:25:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:25:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:54 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:25:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:54 smithi094 ceph-mon[40484]: pgmap v798: 292 pgs: 14 creating+peering, 278 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:54 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:25:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:54 smithi094 ceph-mon[40484]: osdmap e520: 8 total, 8 up, 8 in 2024-03-12T03:25:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:25:54.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:54 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:25:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:54 smithi151 ceph-mon[32923]: pgmap v798: 292 pgs: 14 creating+peering, 278 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:54 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:25:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:54 smithi151 ceph-mon[32923]: osdmap e520: 8 total, 8 up, 8 in 2024-03-12T03:25:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:25:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:54 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:25:55.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:55 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:25:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:55 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-12T03:25:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:55 smithi094 ceph-mon[29291]: osdmap e521: 8 total, 8 up, 8 in 2024-03-12T03:25:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-12T03:25:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:55 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-12T03:25:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:55 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:25:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:55 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-12T03:25:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:55 smithi094 ceph-mon[40484]: osdmap e521: 8 total, 8 up, 8 in 2024-03-12T03:25:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-12T03:25:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:55 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-12T03:25:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:55 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:25:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:55 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-12T03:25:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:55 smithi151 ceph-mon[32923]: osdmap e521: 8 total, 8 up, 8 in 2024-03-12T03:25:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-12T03:25:55.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:55 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-12T03:25:56.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:56 smithi094 ceph-mon[40484]: pgmap v801: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:56 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-12T03:25:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:56 smithi094 ceph-mon[40484]: osdmap e522: 8 total, 8 up, 8 in 2024-03-12T03:25:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:56 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:56 smithi094 ceph-mon[29291]: pgmap v801: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:56 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-12T03:25:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:56 smithi094 ceph-mon[29291]: osdmap e522: 8 total, 8 up, 8 in 2024-03-12T03:25:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:56 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:56 smithi151 ceph-mon[32923]: pgmap v801: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:25:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:56 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-12T03:25:56.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:56 smithi151 ceph-mon[32923]: osdmap e522: 8 total, 8 up, 8 in 2024-03-12T03:25:56.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:56.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:56 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:25:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:57 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:25:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:57 smithi094 ceph-mon[29291]: osdmap e523: 8 total, 8 up, 8 in 2024-03-12T03:25:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100"}]: dispatch 2024-03-12T03:25:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:57 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100"}]: dispatch 2024-03-12T03:25:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:57 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100"}]': finished 2024-03-12T03:25:57.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:57 smithi094 ceph-mon[29291]: osdmap e524: 8 total, 8 up, 8 in 2024-03-12T03:25:57.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:57 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:25:57.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:57 smithi094 ceph-mon[40484]: osdmap e523: 8 total, 8 up, 8 in 2024-03-12T03:25:57.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100"}]: dispatch 2024-03-12T03:25:57.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:57 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100"}]: dispatch 2024-03-12T03:25:57.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:57 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100"}]': finished 2024-03-12T03:25:57.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:57 smithi094 ceph-mon[40484]: osdmap e524: 8 total, 8 up, 8 in 2024-03-12T03:25:57.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:57 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]': finished 2024-03-12T03:25:57.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:57 smithi151 ceph-mon[32923]: osdmap e523: 8 total, 8 up, 8 in 2024-03-12T03:25:57.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100"}]: dispatch 2024-03-12T03:25:57.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:57 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100"}]: dispatch 2024-03-12T03:25:57.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:57 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-100"}]': finished 2024-03-12T03:25:57.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:57 smithi151 ceph-mon[32923]: osdmap e524: 8 total, 8 up, 8 in 2024-03-12T03:25:58.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:25:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:25:58.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:58 smithi094 ceph-mon[29291]: pgmap v804: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:25:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:58 smithi094 ceph-mon[40484]: pgmap v804: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:25:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:58 smithi151 ceph-mon[32923]: pgmap v804: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:25:59.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:59.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:25:59 smithi094 ceph-mon[29291]: osdmap e525: 8 total, 8 up, 8 in 2024-03-12T03:25:59.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:59.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:25:59 smithi094 ceph-mon[40484]: osdmap e525: 8 total, 8 up, 8 in 2024-03-12T03:25:59.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:25:59.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:25:59 smithi151 ceph-mon[32923]: osdmap e525: 8 total, 8 up, 8 in 2024-03-12T03:26:00.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:26:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:26:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:26:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:00 smithi094 ceph-mon[29291]: pgmap v807: 260 pgs: 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:26:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:00 smithi094 ceph-mon[29291]: osdmap e526: 8 total, 8 up, 8 in 2024-03-12T03:26:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:00 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:00 smithi094 ceph-mon[40484]: pgmap v807: 260 pgs: 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:26:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:00 smithi094 ceph-mon[40484]: osdmap e526: 8 total, 8 up, 8 in 2024-03-12T03:26:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:00 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:00 smithi151 ceph-mon[32923]: pgmap v807: 260 pgs: 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:26:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:00 smithi151 ceph-mon[32923]: osdmap e526: 8 total, 8 up, 8 in 2024-03-12T03:26:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:00 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:01.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:01 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:01.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:01 smithi094 ceph-mon[29291]: osdmap e527: 8 total, 8 up, 8 in 2024-03-12T03:26:01.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:26:01.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:26:01.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:01 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:26:01.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:01 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:01.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:01 smithi094 ceph-mon[40484]: osdmap e527: 8 total, 8 up, 8 in 2024-03-12T03:26:01.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:26:01.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:26:01.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:01 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:26:01.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:01 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:01.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:01 smithi151 ceph-mon[32923]: osdmap e527: 8 total, 8 up, 8 in 2024-03-12T03:26:01.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-12T03:26:01.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:26:01.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:01 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-12T03:26:02.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:02 smithi094 ceph-mon[29291]: pgmap v810: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:26:02.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:02 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:26:02.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:02 smithi094 ceph-mon[29291]: osdmap e528: 8 total, 8 up, 8 in 2024-03-12T03:26:02.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:26:02.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:02 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:26:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:02 smithi094 ceph-mon[40484]: pgmap v810: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:26:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:02 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:26:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:02 smithi094 ceph-mon[40484]: osdmap e528: 8 total, 8 up, 8 in 2024-03-12T03:26:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:26:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:02 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:26:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:02 smithi151 ceph-mon[32923]: pgmap v810: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:26:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:02 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-12T03:26:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:02 smithi151 ceph-mon[32923]: osdmap e528: 8 total, 8 up, 8 in 2024-03-12T03:26:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:26:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:02 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-12T03:26:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:03 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:26:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:03 smithi094 ceph-mon[29291]: osdmap e529: 8 total, 8 up, 8 in 2024-03-12T03:26:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:26:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:03 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:26:03.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:03 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:26:03.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:03 smithi094 ceph-mon[29291]: osdmap e530: 8 total, 8 up, 8 in 2024-03-12T03:26:03.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:03 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:26:03.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:03 smithi094 ceph-mon[40484]: osdmap e529: 8 total, 8 up, 8 in 2024-03-12T03:26:03.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:26:03.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:03 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:26:03.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:03 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:26:03.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:03 smithi094 ceph-mon[40484]: osdmap e530: 8 total, 8 up, 8 in 2024-03-12T03:26:03.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:03 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-12T03:26:03.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:03 smithi151 ceph-mon[32923]: osdmap e529: 8 total, 8 up, 8 in 2024-03-12T03:26:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:26:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:03 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-12T03:26:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:03 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-12T03:26:03.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:03 smithi151 ceph-mon[32923]: osdmap e530: 8 total, 8 up, 8 in 2024-03-12T03:26:04.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:04 smithi094 ceph-mon[40484]: pgmap v813: 292 pgs: 18 unknown, 274 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:26:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:04 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:26:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-102"}]: dispatch 2024-03-12T03:26:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:04 smithi094 ceph-mon[40484]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-102"}]: dispatch 2024-03-12T03:26:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:26:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:04 smithi094 ceph-mon[29291]: pgmap v813: 292 pgs: 18 unknown, 274 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:26:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:04 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:26:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-102"}]: dispatch 2024-03-12T03:26:04.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:04 smithi094 ceph-mon[29291]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-102"}]: dispatch 2024-03-12T03:26:04.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:26:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:04 smithi151 ceph-mon[32923]: pgmap v813: 292 pgs: 18 unknown, 274 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:26:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:04 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-6"}]: dispatch 2024-03-12T03:26:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1594320548' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-102"}]: dispatch 2024-03-12T03:26:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:04 smithi151 ceph-mon[32923]: from='client.29353 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-6", "tierpool": "test-rados-api-smithi094-78452-102"}]: dispatch 2024-03-12T03:26:04.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:26:05.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:05 smithi094 ceph-mon[40484]: osdmap e531: 8 total, 8 up, 8 in 2024-03-12T03:26:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:05 smithi094 ceph-mon[29291]: osdmap e531: 8 total, 8 up, 8 in 2024-03-12T03:26:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:05 smithi151 ceph-mon[32923]: osdmap e531: 8 total, 8 up, 8 in 2024-03-12T03:26:06.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[40484]: pgmap v816: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[40484]: osdmap e532: 8 total, 8 up, 8 in 2024-03-12T03:26:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[40484]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[40484]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[40484]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[29291]: pgmap v816: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[29291]: osdmap e532: 8 total, 8 up, 8 in 2024-03-12T03:26:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:06.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[29291]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:06.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:06.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[29291]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:06.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:06.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:06 smithi094 ceph-mon[29291]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:06 smithi151 ceph-mon[32923]: pgmap v816: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:06 smithi151 ceph-mon[32923]: osdmap e532: 8 total, 8 up, 8 in 2024-03-12T03:26:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:06 smithi151 ceph-mon[32923]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:06 smithi151 ceph-mon[32923]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:06.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:06 smithi151 ceph-mon[32923]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:07.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:07 smithi094 ceph-mon[29291]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:26:07.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:07 smithi094 ceph-mon[29291]: osdmap e533: 8 total, 8 up, 8 in 2024-03-12T03:26:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi094-78452-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:07 smithi094 ceph-mon[29291]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi094-78452-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:07 smithi094 ceph-mon[40484]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:26:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:07 smithi094 ceph-mon[40484]: osdmap e533: 8 total, 8 up, 8 in 2024-03-12T03:26:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi094-78452-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:07 smithi094 ceph-mon[40484]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi094-78452-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:07 smithi151 ceph-mon[32923]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:26:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:07 smithi151 ceph-mon[32923]: osdmap e533: 8 total, 8 up, 8 in 2024-03-12T03:26:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi094-78452-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:07 smithi151 ceph-mon[32923]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi094-78452-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:08.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:26:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:26:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:08 smithi094 ceph-mon[29291]: pgmap v819: 228 pgs: 228 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:08 smithi094 ceph-mon[29291]: osdmap e534: 8 total, 8 up, 8 in 2024-03-12T03:26:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:08 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:26:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:08 smithi094 ceph-mon[40484]: pgmap v819: 228 pgs: 228 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:08 smithi094 ceph-mon[40484]: osdmap e534: 8 total, 8 up, 8 in 2024-03-12T03:26:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:08 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:26:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:08 smithi151 ceph-mon[32923]: pgmap v819: 228 pgs: 228 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:08 smithi151 ceph-mon[32923]: osdmap e534: 8 total, 8 up, 8 in 2024-03-12T03:26:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:08 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:26:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:09 smithi094 ceph-mon[40484]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi094-78452-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi094-78452-104"}]': finished 2024-03-12T03:26:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:09 smithi094 ceph-mon[40484]: osdmap e535: 8 total, 8 up, 8 in 2024-03-12T03:26:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:09 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:26:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:09 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:26:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:09 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:26:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:09.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:09 smithi094 ceph-mon[29291]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi094-78452-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi094-78452-104"}]': finished 2024-03-12T03:26:09.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:09 smithi094 ceph-mon[29291]: osdmap e535: 8 total, 8 up, 8 in 2024-03-12T03:26:09.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:09 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:26:09.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:09 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:26:09.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:09 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:26:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:09 smithi151 ceph-mon[32923]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi094-78452-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi094-78452-104"}]': finished 2024-03-12T03:26:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:09 smithi151 ceph-mon[32923]: osdmap e535: 8 total, 8 up, 8 in 2024-03-12T03:26:09.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:09 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:26:09.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:09 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:26:09.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:09 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:26:10.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:10 smithi094 ceph-mon[29291]: pgmap v822: 236 pgs: 8 unknown, 228 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:10.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:10 smithi094 ceph-mon[29291]: osdmap e536: 8 total, 8 up, 8 in 2024-03-12T03:26:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:10 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:10 smithi094 ceph-mon[40484]: pgmap v822: 236 pgs: 8 unknown, 228 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:10 smithi094 ceph-mon[40484]: osdmap e536: 8 total, 8 up, 8 in 2024-03-12T03:26:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:10 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:26:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:26:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:26:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:10 smithi151 ceph-mon[32923]: pgmap v822: 236 pgs: 8 unknown, 228 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:10 smithi151 ceph-mon[32923]: osdmap e536: 8 total, 8 up, 8 in 2024-03-12T03:26:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:10 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:11.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:11 smithi094 ceph-mon[40484]: osdmap e537: 8 total, 8 up, 8 in 2024-03-12T03:26:11.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:11.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:11 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:11 smithi094 ceph-mon[29291]: osdmap e537: 8 total, 8 up, 8 in 2024-03-12T03:26:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:11 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:11.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:11 smithi151 ceph-mon[32923]: osdmap e537: 8 total, 8 up, 8 in 2024-03-12T03:26:11.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:11.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:11 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:12.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:12 smithi094 ceph-mon[29291]: pgmap v825: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:12.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:12 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:12.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:12 smithi094 ceph-mon[29291]: osdmap e538: 8 total, 8 up, 8 in 2024-03-12T03:26:12.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:12 smithi094 ceph-mon[40484]: pgmap v825: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:12.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:12 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:12.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:12 smithi094 ceph-mon[40484]: osdmap e538: 8 total, 8 up, 8 in 2024-03-12T03:26:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:12 smithi151 ceph-mon[32923]: pgmap v825: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:12 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:12 smithi151 ceph-mon[32923]: osdmap e538: 8 total, 8 up, 8 in 2024-03-12T03:26:13.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:13 smithi094 ceph-mon[40484]: osdmap e539: 8 total, 8 up, 8 in 2024-03-12T03:26:13.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:13.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:13 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:13.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:13 smithi094 ceph-mon[29291]: osdmap e539: 8 total, 8 up, 8 in 2024-03-12T03:26:13.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:13.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:13 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:13.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:13 smithi151 ceph-mon[32923]: osdmap e539: 8 total, 8 up, 8 in 2024-03-12T03:26:13.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:13.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:13 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:14 smithi151 ceph-mon[32923]: pgmap v828: 300 pgs: 14 creating+peering, 38 unknown, 248 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:26:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:14 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:14 smithi151 ceph-mon[32923]: osdmap e540: 8 total, 8 up, 8 in 2024-03-12T03:26:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:14.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:14 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:14 smithi094 ceph-mon[29291]: pgmap v828: 300 pgs: 14 creating+peering, 38 unknown, 248 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:26:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:14 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:14 smithi094 ceph-mon[29291]: osdmap e540: 8 total, 8 up, 8 in 2024-03-12T03:26:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:14 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:14.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:14 smithi094 ceph-mon[40484]: pgmap v828: 300 pgs: 14 creating+peering, 38 unknown, 248 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:26:14.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:14 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:14.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:14 smithi094 ceph-mon[40484]: osdmap e540: 8 total, 8 up, 8 in 2024-03-12T03:26:14.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:14.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:14 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:15.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:15 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]': finished 2024-03-12T03:26:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:15 smithi094 ceph-mon[40484]: osdmap e541: 8 total, 8 up, 8 in 2024-03-12T03:26:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:15 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-107", "overlaypool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:15 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-107", "overlaypool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:15 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]': finished 2024-03-12T03:26:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:15 smithi094 ceph-mon[29291]: osdmap e541: 8 total, 8 up, 8 in 2024-03-12T03:26:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:15 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-107", "overlaypool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:15 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-107", "overlaypool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:15 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]': finished 2024-03-12T03:26:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:15 smithi151 ceph-mon[32923]: osdmap e541: 8 total, 8 up, 8 in 2024-03-12T03:26:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:15 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-107", "overlaypool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:15 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-107", "overlaypool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:16 smithi094 ceph-mon[29291]: pgmap v831: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:26:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:16 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:16 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-107", "overlaypool": "test-rados-api-smithi094-78452-107-cache"}]': finished 2024-03-12T03:26:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:16 smithi094 ceph-mon[29291]: osdmap e542: 8 total, 8 up, 8 in 2024-03-12T03:26:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-107-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:26:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:16 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-107-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:26:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:16 smithi094 ceph-mon[40484]: pgmap v831: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:26:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:16 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:16.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:16 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-107", "overlaypool": "test-rados-api-smithi094-78452-107-cache"}]': finished 2024-03-12T03:26:16.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:16 smithi094 ceph-mon[40484]: osdmap e542: 8 total, 8 up, 8 in 2024-03-12T03:26:16.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-107-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:26:16.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:16 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-107-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:26:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:16 smithi151 ceph-mon[32923]: pgmap v831: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:26:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:16 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:16 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-107", "overlaypool": "test-rados-api-smithi094-78452-107-cache"}]': finished 2024-03-12T03:26:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:16 smithi151 ceph-mon[32923]: osdmap e542: 8 total, 8 up, 8 in 2024-03-12T03:26:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-107-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:26:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:16 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-107-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:26:17.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:17 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:26:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:17 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-107-cache", "mode": "writeback"}]': finished 2024-03-12T03:26:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:17 smithi094 ceph-mon[29291]: osdmap e543: 8 total, 8 up, 8 in 2024-03-12T03:26:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-107"}]: dispatch 2024-03-12T03:26:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:17 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-107"}]: dispatch 2024-03-12T03:26:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:17 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:26:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:17 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-107-cache", "mode": "writeback"}]': finished 2024-03-12T03:26:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:17 smithi094 ceph-mon[40484]: osdmap e543: 8 total, 8 up, 8 in 2024-03-12T03:26:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-107"}]: dispatch 2024-03-12T03:26:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:17 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-107"}]: dispatch 2024-03-12T03:26:17.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:17 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:26:17.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:17 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-107-cache", "mode": "writeback"}]': finished 2024-03-12T03:26:17.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:17 smithi151 ceph-mon[32923]: osdmap e543: 8 total, 8 up, 8 in 2024-03-12T03:26:17.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-107"}]: dispatch 2024-03-12T03:26:17.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:17 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-107"}]: dispatch 2024-03-12T03:26:18.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:26:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:26:18.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:18 smithi094 ceph-mon[40484]: pgmap v834: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:18 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-107"}]': finished 2024-03-12T03:26:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:18 smithi094 ceph-mon[40484]: osdmap e544: 8 total, 8 up, 8 in 2024-03-12T03:26:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:18 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:18 smithi094 ceph-mon[29291]: pgmap v834: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:18 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-107"}]': finished 2024-03-12T03:26:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:18 smithi094 ceph-mon[29291]: osdmap e544: 8 total, 8 up, 8 in 2024-03-12T03:26:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:18 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:18 smithi151 ceph-mon[32923]: pgmap v834: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:18 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-107"}]': finished 2024-03-12T03:26:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:18 smithi151 ceph-mon[32923]: osdmap e544: 8 total, 8 up, 8 in 2024-03-12T03:26:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1468286132' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:18 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]: dispatch 2024-03-12T03:26:19.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:19.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:19 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:26:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:19 smithi094 ceph-mon[40484]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]': finished 2024-03-12T03:26:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:19 smithi094 ceph-mon[40484]: osdmap e545: 8 total, 8 up, 8 in 2024-03-12T03:26:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:26:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:19 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:26:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:19 smithi094 ceph-mon[29291]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]': finished 2024-03-12T03:26:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:19 smithi094 ceph-mon[29291]: osdmap e545: 8 total, 8 up, 8 in 2024-03-12T03:26:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:26:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:19 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:26:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:19 smithi151 ceph-mon[32923]: from='client.57062 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-107", "tierpool": "test-rados-api-smithi094-78452-107-cache"}]': finished 2024-03-12T03:26:19.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:19 smithi151 ceph-mon[32923]: osdmap e545: 8 total, 8 up, 8 in 2024-03-12T03:26:19.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:26:20.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:20 smithi094 ceph-mon[29291]: pgmap v837: 300 pgs: 300 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:26:20.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:20 smithi094 ceph-mon[29291]: osdmap e546: 8 total, 8 up, 8 in 2024-03-12T03:26:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:20 smithi094 ceph-mon[40484]: pgmap v837: 300 pgs: 300 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:26:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:20 smithi094 ceph-mon[40484]: osdmap e546: 8 total, 8 up, 8 in 2024-03-12T03:26:20.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:26:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:26:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:26:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:20 smithi151 ceph-mon[32923]: pgmap v837: 300 pgs: 300 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:26:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:20 smithi151 ceph-mon[32923]: osdmap e546: 8 total, 8 up, 8 in 2024-03-12T03:26:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[40484]: osdmap e547: 8 total, 8 up, 8 in 2024-03-12T03:26:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[29291]: osdmap e547: 8 total, 8 up, 8 in 2024-03-12T03:26:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:21.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:21.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:21.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:21.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:21.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:21 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:21 smithi151 ceph-mon[32923]: osdmap e547: 8 total, 8 up, 8 in 2024-03-12T03:26:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:21 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:21 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:21 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:23.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:22 smithi094 ceph-mon[40484]: pgmap v840: 236 pgs: 236 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:23.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:22 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:26:23.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:22 smithi094 ceph-mon[40484]: osdmap e548: 8 total, 8 up, 8 in 2024-03-12T03:26:23.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:23.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:22 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:23.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:22 smithi094 ceph-mon[29291]: pgmap v840: 236 pgs: 236 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:23.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:22 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:26:23.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:22 smithi094 ceph-mon[29291]: osdmap e548: 8 total, 8 up, 8 in 2024-03-12T03:26:23.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:23.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:22 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:23.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:22 smithi151 ceph-mon[32923]: pgmap v840: 236 pgs: 236 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:23.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:22 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:26:23.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:22 smithi151 ceph-mon[32923]: osdmap e548: 8 total, 8 up, 8 in 2024-03-12T03:26:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:23.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:22 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:24.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:23 smithi094 ceph-mon[40484]: pgmap v843: 236 pgs: 236 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:24.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:23 smithi094 ceph-mon[40484]: osdmap e549: 8 total, 8 up, 8 in 2024-03-12T03:26:24.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:23 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-109"}]': finished 2024-03-12T03:26:24.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:23 smithi094 ceph-mon[40484]: osdmap e550: 8 total, 8 up, 8 in 2024-03-12T03:26:24.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:23 smithi094 ceph-mon[29291]: pgmap v843: 236 pgs: 236 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:24.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:23 smithi094 ceph-mon[29291]: osdmap e549: 8 total, 8 up, 8 in 2024-03-12T03:26:24.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:23 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-109"}]': finished 2024-03-12T03:26:24.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:23 smithi094 ceph-mon[29291]: osdmap e550: 8 total, 8 up, 8 in 2024-03-12T03:26:24.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:23 smithi151 ceph-mon[32923]: pgmap v843: 236 pgs: 236 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:24.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:23 smithi151 ceph-mon[32923]: osdmap e549: 8 total, 8 up, 8 in 2024-03-12T03:26:24.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:23 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-109"}]': finished 2024-03-12T03:26:24.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:23 smithi151 ceph-mon[32923]: osdmap e550: 8 total, 8 up, 8 in 2024-03-12T03:26:25.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:25 smithi094 ceph-mon[29291]: osdmap e551: 8 total, 8 up, 8 in 2024-03-12T03:26:25.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:25.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:25 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:25.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:25 smithi094 ceph-mon[29291]: pgmap v846: 276 pgs: 40 unknown, 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:25.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:25 smithi094 ceph-mon[40484]: osdmap e551: 8 total, 8 up, 8 in 2024-03-12T03:26:25.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:25.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:25 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:25.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:25 smithi094 ceph-mon[40484]: pgmap v846: 276 pgs: 40 unknown, 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:25.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:25 smithi151 ceph-mon[32923]: osdmap e551: 8 total, 8 up, 8 in 2024-03-12T03:26:25.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:25.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:25 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:25.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:25 smithi151 ceph-mon[32923]: pgmap v846: 276 pgs: 40 unknown, 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:26.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:26 smithi094 ceph-mon[29291]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:26.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:26 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:26.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:26 smithi094 ceph-mon[29291]: osdmap e552: 8 total, 8 up, 8 in 2024-03-12T03:26:26.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:26.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:26 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:26 smithi094 ceph-mon[40484]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:26 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:26 smithi094 ceph-mon[40484]: osdmap e552: 8 total, 8 up, 8 in 2024-03-12T03:26:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:26 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:26 smithi151 ceph-mon[32923]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:26 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:26 smithi151 ceph-mon[32923]: osdmap e552: 8 total, 8 up, 8 in 2024-03-12T03:26:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:26 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:27.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:27 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]': finished 2024-03-12T03:26:27.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:27 smithi094 ceph-mon[29291]: osdmap e553: 8 total, 8 up, 8 in 2024-03-12T03:26:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-109", "overlaypool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:27 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-109", "overlaypool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:27 smithi094 ceph-mon[29291]: pgmap v849: 276 pgs: 40 unknown, 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:27 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]': finished 2024-03-12T03:26:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:27 smithi094 ceph-mon[40484]: osdmap e553: 8 total, 8 up, 8 in 2024-03-12T03:26:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-109", "overlaypool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:27 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-109", "overlaypool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:27 smithi094 ceph-mon[40484]: pgmap v849: 276 pgs: 40 unknown, 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:27 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]': finished 2024-03-12T03:26:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:27 smithi151 ceph-mon[32923]: osdmap e553: 8 total, 8 up, 8 in 2024-03-12T03:26:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-109", "overlaypool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:27 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-109", "overlaypool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:27 smithi151 ceph-mon[32923]: pgmap v849: 276 pgs: 40 unknown, 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:28.674 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:26:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:26:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:28 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-109", "overlaypool": "test-rados-api-smithi094-78452-109-cache"}]': finished 2024-03-12T03:26:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:28 smithi094 ceph-mon[29291]: osdmap e554: 8 total, 8 up, 8 in 2024-03-12T03:26:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-109-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:26:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:28 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-109-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:26:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:28 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:28 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-109", "overlaypool": "test-rados-api-smithi094-78452-109-cache"}]': finished 2024-03-12T03:26:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:28 smithi094 ceph-mon[40484]: osdmap e554: 8 total, 8 up, 8 in 2024-03-12T03:26:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-109-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:26:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:28 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-109-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:26:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:28 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:28 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-109", "overlaypool": "test-rados-api-smithi094-78452-109-cache"}]': finished 2024-03-12T03:26:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:28 smithi151 ceph-mon[32923]: osdmap e554: 8 total, 8 up, 8 in 2024-03-12T03:26:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-109-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:26:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:28 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-109-cache", "mode": "writeback"}]: dispatch 2024-03-12T03:26:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:28 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:29.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:29 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:26:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:29 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-109-cache", "mode": "writeback"}]': finished 2024-03-12T03:26:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:29 smithi094 ceph-mon[29291]: osdmap e555: 8 total, 8 up, 8 in 2024-03-12T03:26:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:26:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:29 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:26:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:29 smithi094 ceph-mon[29291]: pgmap v852: 276 pgs: 276 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:29 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:26:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:29 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-109-cache", "mode": "writeback"}]': finished 2024-03-12T03:26:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:29 smithi094 ceph-mon[40484]: osdmap e555: 8 total, 8 up, 8 in 2024-03-12T03:26:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:26:29.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:29 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:26:29.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:29 smithi094 ceph-mon[40484]: pgmap v852: 276 pgs: 276 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:29 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:26:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:29 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-109-cache", "mode": "writeback"}]': finished 2024-03-12T03:26:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:29 smithi151 ceph-mon[32923]: osdmap e555: 8 total, 8 up, 8 in 2024-03-12T03:26:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:26:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:29 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:26:29.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:29 smithi151 ceph-mon[32923]: pgmap v852: 276 pgs: 276 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:30.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:30 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:26:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:30 smithi094 ceph-mon[40484]: osdmap e556: 8 total, 8 up, 8 in 2024-03-12T03:26:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:26:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:30 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:26:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:30 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:26:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:30 smithi094 ceph-mon[29291]: osdmap e556: 8 total, 8 up, 8 in 2024-03-12T03:26:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:26:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:30 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:26:30.961 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:26:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:26:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:26:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:30 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:26:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:30 smithi151 ceph-mon[32923]: osdmap e556: 8 total, 8 up, 8 in 2024-03-12T03:26:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:26:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:30 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:26:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:26:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[40484]: osdmap e557: 8 total, 8 up, 8 in 2024-03-12T03:26:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:26:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:26:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[40484]: pgmap v855: 276 pgs: 276 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:26:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-12T03:26:31.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[40484]: osdmap e558: 8 total, 8 up, 8 in 2024-03-12T03:26:31.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:26:31.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[29291]: osdmap e557: 8 total, 8 up, 8 in 2024-03-12T03:26:31.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:26:31.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:26:31.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[29291]: pgmap v855: 276 pgs: 276 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:31.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:26:31.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-12T03:26:31.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:31 smithi094 ceph-mon[29291]: osdmap e558: 8 total, 8 up, 8 in 2024-03-12T03:26:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:31 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:26:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:31 smithi151 ceph-mon[32923]: osdmap e557: 8 total, 8 up, 8 in 2024-03-12T03:26:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:26:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:31 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:26:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:31 smithi151 ceph-mon[32923]: pgmap v855: 276 pgs: 276 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:31.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:31 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:26:31.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:31 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-12T03:26:31.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:31 smithi151 ceph-mon[32923]: osdmap e558: 8 total, 8 up, 8 in 2024-03-12T03:26:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-12T03:26:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:32 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-12T03:26:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:32 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-12T03:26:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:32 smithi094 ceph-mon[29291]: osdmap e559: 8 total, 8 up, 8 in 2024-03-12T03:26:32.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-12T03:26:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:32 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-12T03:26:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:32 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-12T03:26:32.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:32 smithi094 ceph-mon[40484]: osdmap e559: 8 total, 8 up, 8 in 2024-03-12T03:26:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-12T03:26:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:32 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-12T03:26:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:32 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-12T03:26:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:32 smithi151 ceph-mon[32923]: osdmap e559: 8 total, 8 up, 8 in 2024-03-12T03:26:33.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:33.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:33 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:33.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:33 smithi094 ceph-mon[29291]: pgmap v858: 276 pgs: 276 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:33.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:33 smithi094 ceph-mon[29291]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:33 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:33 smithi094 ceph-mon[40484]: pgmap v858: 276 pgs: 276 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:33 smithi094 ceph-mon[40484]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:33.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:33.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:33 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:33.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:33 smithi151 ceph-mon[32923]: pgmap v858: 276 pgs: 276 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:33.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:33 smithi151 ceph-mon[32923]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:34.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:34 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-109"}]': finished 2024-03-12T03:26:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:34 smithi094 ceph-mon[40484]: osdmap e560: 8 total, 8 up, 8 in 2024-03-12T03:26:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:34 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:26:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:34 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-109"}]': finished 2024-03-12T03:26:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:34 smithi094 ceph-mon[29291]: osdmap e560: 8 total, 8 up, 8 in 2024-03-12T03:26:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:34 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:26:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:34 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-109"}]': finished 2024-03-12T03:26:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:34 smithi151 ceph-mon[32923]: osdmap e560: 8 total, 8 up, 8 in 2024-03-12T03:26:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:34 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]: dispatch 2024-03-12T03:26:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:26:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:35 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]': finished 2024-03-12T03:26:35.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:35 smithi151 ceph-mon[32923]: osdmap e561: 8 total, 8 up, 8 in 2024-03-12T03:26:35.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:35 smithi151 ceph-mon[32923]: pgmap v861: 276 pgs: 276 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:36.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:35 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]': finished 2024-03-12T03:26:36.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:35 smithi094 ceph-mon[29291]: osdmap e561: 8 total, 8 up, 8 in 2024-03-12T03:26:36.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:35 smithi094 ceph-mon[29291]: pgmap v861: 276 pgs: 276 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:35 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-109", "tierpool": "test-rados-api-smithi094-78452-109-cache"}]': finished 2024-03-12T03:26:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:35 smithi094 ceph-mon[40484]: osdmap e561: 8 total, 8 up, 8 in 2024-03-12T03:26:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:35 smithi094 ceph-mon[40484]: pgmap v861: 276 pgs: 276 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:37.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:36 smithi094 ceph-mon[29291]: osdmap e562: 8 total, 8 up, 8 in 2024-03-12T03:26:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:36 smithi094 ceph-mon[40484]: osdmap e562: 8 total, 8 up, 8 in 2024-03-12T03:26:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:36 smithi151 ceph-mon[32923]: osdmap e562: 8 total, 8 up, 8 in 2024-03-12T03:26:38.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:37 smithi094 ceph-mon[29291]: osdmap e563: 8 total, 8 up, 8 in 2024-03-12T03:26:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:37 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:37 smithi094 ceph-mon[29291]: pgmap v864: 236 pgs: 236 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:37 smithi094 ceph-mon[40484]: osdmap e563: 8 total, 8 up, 8 in 2024-03-12T03:26:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:37 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:37 smithi094 ceph-mon[40484]: pgmap v864: 236 pgs: 236 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:38.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:37 smithi151 ceph-mon[32923]: osdmap e563: 8 total, 8 up, 8 in 2024-03-12T03:26:38.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:38.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:37 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:38.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:37 smithi151 ceph-mon[32923]: pgmap v864: 236 pgs: 236 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:38.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:26:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:26:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:38 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]': finished 2024-03-12T03:26:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:38 smithi094 ceph-mon[29291]: osdmap e564: 8 total, 8 up, 8 in 2024-03-12T03:26:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:38 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:38 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:38 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:38 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]': finished 2024-03-12T03:26:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:38 smithi094 ceph-mon[40484]: osdmap e564: 8 total, 8 up, 8 in 2024-03-12T03:26:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:39.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:38 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:39.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:38 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:39.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:38 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:39.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:38 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-109"}]': finished 2024-03-12T03:26:39.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:38 smithi151 ceph-mon[32923]: osdmap e564: 8 total, 8 up, 8 in 2024-03-12T03:26:39.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/753741023' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:39.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:38 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]: dispatch 2024-03-12T03:26:39.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:38 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:39.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:38 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:40.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:39 smithi094 ceph-mon[29291]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]': finished 2024-03-12T03:26:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:39 smithi094 ceph-mon[29291]: osdmap e565: 8 total, 8 up, 8 in 2024-03-12T03:26:40.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:39 smithi094 ceph-mon[29291]: pgmap v867: 236 pgs: 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:40.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:39 smithi094 ceph-mon[40484]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]': finished 2024-03-12T03:26:40.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:39 smithi094 ceph-mon[40484]: osdmap e565: 8 total, 8 up, 8 in 2024-03-12T03:26:40.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:39 smithi094 ceph-mon[40484]: pgmap v867: 236 pgs: 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:40.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:39 smithi151 ceph-mon[32923]: from='client.57068 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-109"}]': finished 2024-03-12T03:26:40.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:39 smithi151 ceph-mon[32923]: osdmap e565: 8 total, 8 up, 8 in 2024-03-12T03:26:40.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:39 smithi151 ceph-mon[32923]: pgmap v867: 236 pgs: 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:40.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:40 smithi094 ceph-mon[29291]: osdmap e566: 8 total, 8 up, 8 in 2024-03-12T03:26:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:40 smithi094 ceph-mon[29291]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:26:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:26:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:26:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:40 smithi094 ceph-mon[40484]: osdmap e566: 8 total, 8 up, 8 in 2024-03-12T03:26:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:40 smithi094 ceph-mon[40484]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:40 smithi151 ceph-mon[32923]: osdmap e566: 8 total, 8 up, 8 in 2024-03-12T03:26:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:41.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:40 smithi151 ceph-mon[32923]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:41 smithi094 ceph-mon[29291]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]': finished 2024-03-12T03:26:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:41 smithi094 ceph-mon[29291]: osdmap e567: 8 total, 8 up, 8 in 2024-03-12T03:26:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:41 smithi094 ceph-mon[29291]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:41 smithi094 ceph-mon[29291]: pgmap v870: 228 pgs: 228 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-12T03:26:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:41 smithi094 ceph-mon[29291]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]': finished 2024-03-12T03:26:42.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:41 smithi094 ceph-mon[40484]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]': finished 2024-03-12T03:26:42.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:41 smithi094 ceph-mon[40484]: osdmap e567: 8 total, 8 up, 8 in 2024-03-12T03:26:42.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:42.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:41 smithi094 ceph-mon[40484]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:42.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:41 smithi094 ceph-mon[40484]: pgmap v870: 228 pgs: 228 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-12T03:26:42.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:41 smithi094 ceph-mon[40484]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]': finished 2024-03-12T03:26:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:41 smithi151 ceph-mon[32923]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi094-78452-104"}]': finished 2024-03-12T03:26:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:41 smithi151 ceph-mon[32923]: osdmap e567: 8 total, 8 up, 8 in 2024-03-12T03:26:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4017526384' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:41 smithi151 ceph-mon[32923]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]: dispatch 2024-03-12T03:26:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:41 smithi151 ceph-mon[32923]: pgmap v870: 228 pgs: 228 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-12T03:26:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:41 smithi151 ceph-mon[32923]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi094-78452-104"}]': finished 2024-03-12T03:26:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:42 smithi094 ceph-mon[29291]: osdmap e568: 8 total, 8 up, 8 in 2024-03-12T03:26:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:42 smithi094 ceph-mon[40484]: osdmap e568: 8 total, 8 up, 8 in 2024-03-12T03:26:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:42 smithi151 ceph-mon[32923]: osdmap e568: 8 total, 8 up, 8 in 2024-03-12T03:26:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-12T03:26:44.115 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:43 smithi151 ceph-mon[32923]: pgmap v872: 228 pgs: 228 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-12T03:26:44.115 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:26:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:43 smithi151 ceph-mon[32923]: osdmap e569: 8 total, 8 up, 8 in 2024-03-12T03:26:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:44.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:43 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:44.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:43 smithi094 ceph-mon[40484]: pgmap v872: 228 pgs: 228 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-12T03:26:44.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:26:44.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:43 smithi094 ceph-mon[40484]: osdmap e569: 8 total, 8 up, 8 in 2024-03-12T03:26:44.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:44.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:43 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:43 smithi094 ceph-mon[29291]: pgmap v872: 228 pgs: 228 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-12T03:26:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi094-78452-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-12T03:26:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:43 smithi094 ceph-mon[29291]: osdmap e569: 8 total, 8 up, 8 in 2024-03-12T03:26:44.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:44.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:43 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:45.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:44 smithi094 ceph-mon[29291]: osdmap e570: 8 total, 8 up, 8 in 2024-03-12T03:26:45.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:44 smithi094 ceph-mon[40484]: osdmap e570: 8 total, 8 up, 8 in 2024-03-12T03:26:45.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:44 smithi151 ceph-mon[32923]: osdmap e570: 8 total, 8 up, 8 in 2024-03-12T03:26:46.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:45 smithi094 ceph-mon[40484]: pgmap v875: 228 pgs: 228 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:46.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:26:46.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:45 smithi094 ceph-mon[40484]: osdmap e571: 8 total, 8 up, 8 in 2024-03-12T03:26:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:45 smithi094 ceph-mon[29291]: pgmap v875: 228 pgs: 228 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:26:46.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:45 smithi094 ceph-mon[29291]: osdmap e571: 8 total, 8 up, 8 in 2024-03-12T03:26:46.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:45 smithi151 ceph-mon[32923]: pgmap v875: 228 pgs: 228 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:26:46.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi094-78452-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:26:46.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:45 smithi151 ceph-mon[32923]: osdmap e571: 8 total, 8 up, 8 in 2024-03-12T03:26:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:46 smithi094 ceph-mon[29291]: osdmap e572: 8 total, 8 up, 8 in 2024-03-12T03:26:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:46 smithi094 ceph-mon[40484]: osdmap e572: 8 total, 8 up, 8 in 2024-03-12T03:26:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:46 smithi151 ceph-mon[32923]: osdmap e572: 8 total, 8 up, 8 in 2024-03-12T03:26:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:48.111 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:47 smithi094 ceph-mon[40484]: pgmap v878: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:48.111 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:47 smithi094 ceph-mon[40484]: osdmap e573: 8 total, 8 up, 8 in 2024-03-12T03:26:48.111 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:26:48.111 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:26:48.111 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:47 smithi094 ceph-mon[40484]: osdmap e574: 8 total, 8 up, 8 in 2024-03-12T03:26:48.111 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-112"}]: dispatch 2024-03-12T03:26:48.112 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:47 smithi094 ceph-mon[29291]: pgmap v878: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:48.112 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:47 smithi094 ceph-mon[29291]: osdmap e573: 8 total, 8 up, 8 in 2024-03-12T03:26:48.112 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:26:48.112 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:26:48.112 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:47 smithi094 ceph-mon[29291]: osdmap e574: 8 total, 8 up, 8 in 2024-03-12T03:26:48.112 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-112"}]: dispatch 2024-03-12T03:26:48.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:47 smithi151 ceph-mon[32923]: pgmap v878: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:48.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:47 smithi151 ceph-mon[32923]: osdmap e573: 8 total, 8 up, 8 in 2024-03-12T03:26:48.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:26:48.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:26:48.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:47 smithi151 ceph-mon[32923]: osdmap e574: 8 total, 8 up, 8 in 2024-03-12T03:26:48.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-112"}]: dispatch 2024-03-12T03:26:48.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:26:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:26:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:48 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:48 smithi094 ceph-mon[29291]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-112"}]': finished 2024-03-12T03:26:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:48 smithi094 ceph-mon[29291]: osdmap e575: 8 total, 8 up, 8 in 2024-03-12T03:26:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:48 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:48 smithi094 ceph-mon[40484]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-112"}]': finished 2024-03-12T03:26:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:48 smithi094 ceph-mon[40484]: osdmap e575: 8 total, 8 up, 8 in 2024-03-12T03:26:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:48 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:48 smithi151 ceph-mon[32923]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-112"}]': finished 2024-03-12T03:26:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:48 smithi151 ceph-mon[32923]: osdmap e575: 8 total, 8 up, 8 in 2024-03-12T03:26:50.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:49 smithi094 ceph-mon[29291]: pgmap v881: 268 pgs: 268 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:26:50.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:50.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:49 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:26:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:49 smithi094 ceph-mon[40484]: pgmap v881: 268 pgs: 268 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:26:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:50.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:49 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:26:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:49 smithi151 ceph-mon[32923]: pgmap v881: 268 pgs: 268 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:26:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:49 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:26:50.817 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:26:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:26:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:26:51.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:26:51.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:50 smithi094 ceph-mon[29291]: osdmap e576: 8 total, 8 up, 8 in 2024-03-12T03:26:51.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-112"}]: dispatch 2024-03-12T03:26:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:26:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:50 smithi094 ceph-mon[40484]: osdmap e576: 8 total, 8 up, 8 in 2024-03-12T03:26:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-112"}]: dispatch 2024-03-12T03:26:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:26:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:50 smithi151 ceph-mon[32923]: osdmap e576: 8 total, 8 up, 8 in 2024-03-12T03:26:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-112"}]: dispatch 2024-03-12T03:26:52.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:52 smithi094 ceph-mon[29291]: pgmap v884: 268 pgs: 268 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:26:52.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-112"}]': finished 2024-03-12T03:26:52.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:52 smithi094 ceph-mon[29291]: osdmap e577: 8 total, 8 up, 8 in 2024-03-12T03:26:52.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:52 smithi094 ceph-mon[40484]: pgmap v884: 268 pgs: 268 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:26:52.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-112"}]': finished 2024-03-12T03:26:52.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:52 smithi094 ceph-mon[40484]: osdmap e577: 8 total, 8 up, 8 in 2024-03-12T03:26:52.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:52 smithi151 ceph-mon[32923]: pgmap v884: 268 pgs: 268 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:26:52.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-112"}]': finished 2024-03-12T03:26:52.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:52 smithi151 ceph-mon[32923]: osdmap e577: 8 total, 8 up, 8 in 2024-03-12T03:26:53.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:53 smithi094 ceph-mon[40484]: osdmap e578: 8 total, 8 up, 8 in 2024-03-12T03:26:53.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:53 smithi094 ceph-mon[40484]: osdmap e579: 8 total, 8 up, 8 in 2024-03-12T03:26:53.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:53.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:53 smithi094 ceph-mon[29291]: osdmap e578: 8 total, 8 up, 8 in 2024-03-12T03:26:53.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:53 smithi094 ceph-mon[29291]: osdmap e579: 8 total, 8 up, 8 in 2024-03-12T03:26:53.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:53.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:53 smithi151 ceph-mon[32923]: osdmap e578: 8 total, 8 up, 8 in 2024-03-12T03:26:53.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:53 smithi151 ceph-mon[32923]: osdmap e579: 8 total, 8 up, 8 in 2024-03-12T03:26:53.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:26:54.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:54 smithi094 ceph-mon[40484]: pgmap v887: 236 pgs: 236 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:26:54.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:54 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:54.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:54.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:54 smithi094 ceph-mon[40484]: osdmap e580: 8 total, 8 up, 8 in 2024-03-12T03:26:54.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:26:54.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:54 smithi094 ceph-mon[29291]: pgmap v887: 236 pgs: 236 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:26:54.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:54 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:54.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:54.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:54 smithi094 ceph-mon[29291]: osdmap e580: 8 total, 8 up, 8 in 2024-03-12T03:26:54.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:26:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:54 smithi151 ceph-mon[32923]: pgmap v887: 236 pgs: 236 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:26:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:54 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:26:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:26:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:54 smithi151 ceph-mon[32923]: osdmap e580: 8 total, 8 up, 8 in 2024-03-12T03:26:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:26:56.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:55 smithi094 ceph-mon[40484]: pgmap v890: 268 pgs: 7 creating+peering, 25 unknown, 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:26:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:26:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:55 smithi094 ceph-mon[40484]: osdmap e581: 8 total, 8 up, 8 in 2024-03-12T03:26:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-114"}]: dispatch 2024-03-12T03:26:56.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:55 smithi094 ceph-mon[29291]: pgmap v890: 268 pgs: 7 creating+peering, 25 unknown, 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:26:56.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:26:56.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:55 smithi094 ceph-mon[29291]: osdmap e581: 8 total, 8 up, 8 in 2024-03-12T03:26:56.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-114"}]: dispatch 2024-03-12T03:26:56.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:55 smithi151 ceph-mon[32923]: pgmap v890: 268 pgs: 7 creating+peering, 25 unknown, 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:26:56.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:26:56.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:55 smithi151 ceph-mon[32923]: osdmap e581: 8 total, 8 up, 8 in 2024-03-12T03:26:56.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-114"}]: dispatch 2024-03-12T03:26:57.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-114"}]': finished 2024-03-12T03:26:57.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:56 smithi094 ceph-mon[29291]: osdmap e582: 8 total, 8 up, 8 in 2024-03-12T03:26:57.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-114", "mode": "writeback"}]: dispatch 2024-03-12T03:26:57.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:56 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:26:57.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-114", "mode": "writeback"}]': finished 2024-03-12T03:26:57.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:56 smithi094 ceph-mon[29291]: osdmap e583: 8 total, 8 up, 8 in 2024-03-12T03:26:57.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-114"}]': finished 2024-03-12T03:26:57.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:56 smithi094 ceph-mon[40484]: osdmap e582: 8 total, 8 up, 8 in 2024-03-12T03:26:57.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-114", "mode": "writeback"}]: dispatch 2024-03-12T03:26:57.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:56 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:26:57.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-114", "mode": "writeback"}]': finished 2024-03-12T03:26:57.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:56 smithi094 ceph-mon[40484]: osdmap e583: 8 total, 8 up, 8 in 2024-03-12T03:26:57.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-114"}]': finished 2024-03-12T03:26:57.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:56 smithi151 ceph-mon[32923]: osdmap e582: 8 total, 8 up, 8 in 2024-03-12T03:26:57.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-114", "mode": "writeback"}]: dispatch 2024-03-12T03:26:57.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:56 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:26:57.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-114", "mode": "writeback"}]': finished 2024-03-12T03:26:57.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:56 smithi151 ceph-mon[32923]: osdmap e583: 8 total, 8 up, 8 in 2024-03-12T03:26:58.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:57 smithi094 ceph-mon[40484]: pgmap v893: 268 pgs: 7 creating+peering, 25 unknown, 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:58.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:58.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:57 smithi094 ceph-mon[29291]: pgmap v893: 268 pgs: 7 creating+peering, 25 unknown, 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:58.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:58.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:57 smithi151 ceph-mon[32923]: pgmap v893: 268 pgs: 7 creating+peering, 25 unknown, 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:26:58.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:26:58.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:26:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:26:59.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:26:59.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:58 smithi094 ceph-mon[40484]: osdmap e584: 8 total, 8 up, 8 in 2024-03-12T03:26:59.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-114"}]: dispatch 2024-03-12T03:26:59.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:26:58 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:59.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:26:59.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:58 smithi094 ceph-mon[29291]: osdmap e584: 8 total, 8 up, 8 in 2024-03-12T03:26:59.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-114"}]: dispatch 2024-03-12T03:26:59.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:26:58 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:26:59.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:26:59.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:58 smithi151 ceph-mon[32923]: osdmap e584: 8 total, 8 up, 8 in 2024-03-12T03:26:59.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-114"}]: dispatch 2024-03-12T03:26:59.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:26:58 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:00.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:00 smithi094 ceph-mon[40484]: pgmap v896: 268 pgs: 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:27:00.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:00 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:00.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-114"}]': finished 2024-03-12T03:27:00.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:00 smithi094 ceph-mon[40484]: osdmap e585: 8 total, 8 up, 8 in 2024-03-12T03:27:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:00 smithi094 ceph-mon[29291]: pgmap v896: 268 pgs: 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:27:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:00 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-114"}]': finished 2024-03-12T03:27:00.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:00 smithi094 ceph-mon[29291]: osdmap e585: 8 total, 8 up, 8 in 2024-03-12T03:27:00.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:00 smithi151 ceph-mon[32923]: pgmap v896: 268 pgs: 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-12T03:27:00.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:00 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:00.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-114"}]': finished 2024-03-12T03:27:00.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:00 smithi151 ceph-mon[32923]: osdmap e585: 8 total, 8 up, 8 in 2024-03-12T03:27:00.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:27:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:27:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:27:01.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:01 smithi094 ceph-mon[40484]: osdmap e586: 8 total, 8 up, 8 in 2024-03-12T03:27:01.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:01 smithi094 ceph-mon[40484]: osdmap e587: 8 total, 8 up, 8 in 2024-03-12T03:27:01.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:01.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:01 smithi094 ceph-mon[29291]: osdmap e586: 8 total, 8 up, 8 in 2024-03-12T03:27:01.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:01 smithi094 ceph-mon[29291]: osdmap e587: 8 total, 8 up, 8 in 2024-03-12T03:27:01.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:01.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:01 smithi151 ceph-mon[32923]: osdmap e586: 8 total, 8 up, 8 in 2024-03-12T03:27:01.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:01 smithi151 ceph-mon[32923]: osdmap e587: 8 total, 8 up, 8 in 2024-03-12T03:27:01.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:02.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:02 smithi094 ceph-mon[29291]: pgmap v899: 236 pgs: 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:02.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:02.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:02 smithi094 ceph-mon[29291]: osdmap e588: 8 total, 8 up, 8 in 2024-03-12T03:27:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:02 smithi094 ceph-mon[40484]: pgmap v899: 236 pgs: 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:02 smithi094 ceph-mon[40484]: osdmap e588: 8 total, 8 up, 8 in 2024-03-12T03:27:02.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:02 smithi151 ceph-mon[32923]: pgmap v899: 236 pgs: 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:02.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:02.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:02 smithi151 ceph-mon[32923]: osdmap e588: 8 total, 8 up, 8 in 2024-03-12T03:27:04.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:03 smithi094 ceph-mon[29291]: pgmap v902: 268 pgs: 22 unknown, 246 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:27:04.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:03 smithi094 ceph-mon[29291]: osdmap e589: 8 total, 8 up, 8 in 2024-03-12T03:27:04.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:04.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:03 smithi094 ceph-mon[40484]: pgmap v902: 268 pgs: 22 unknown, 246 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:27:04.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:03 smithi094 ceph-mon[40484]: osdmap e589: 8 total, 8 up, 8 in 2024-03-12T03:27:04.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:04.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:03 smithi151 ceph-mon[32923]: pgmap v902: 268 pgs: 22 unknown, 246 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:27:04.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:03 smithi151 ceph-mon[32923]: osdmap e589: 8 total, 8 up, 8 in 2024-03-12T03:27:04.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:05.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:05.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[29291]: osdmap e590: 8 total, 8 up, 8 in 2024-03-12T03:27:05.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-116"}]: dispatch 2024-03-12T03:27:05.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:27:05.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-116"}]': finished 2024-03-12T03:27:05.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[29291]: osdmap e591: 8 total, 8 up, 8 in 2024-03-12T03:27:05.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-116", "mode": "writeback"}]: dispatch 2024-03-12T03:27:05.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:05.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[40484]: osdmap e590: 8 total, 8 up, 8 in 2024-03-12T03:27:05.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-116"}]: dispatch 2024-03-12T03:27:05.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:27:05.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-116"}]': finished 2024-03-12T03:27:05.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[40484]: osdmap e591: 8 total, 8 up, 8 in 2024-03-12T03:27:05.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-116", "mode": "writeback"}]: dispatch 2024-03-12T03:27:05.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:05.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:04 smithi151 ceph-mon[32923]: osdmap e590: 8 total, 8 up, 8 in 2024-03-12T03:27:05.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-116"}]: dispatch 2024-03-12T03:27:05.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:04 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:27:05.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-116"}]': finished 2024-03-12T03:27:05.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:04 smithi151 ceph-mon[32923]: osdmap e591: 8 total, 8 up, 8 in 2024-03-12T03:27:05.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-116", "mode": "writeback"}]: dispatch 2024-03-12T03:27:06.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:05 smithi094 ceph-mon[29291]: pgmap v905: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:27:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:05 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-116", "mode": "writeback"}]': finished 2024-03-12T03:27:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:05 smithi094 ceph-mon[29291]: osdmap e592: 8 total, 8 up, 8 in 2024-03-12T03:27:06.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:05 smithi094 ceph-mon[40484]: pgmap v905: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:27:06.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:05 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:06.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-116", "mode": "writeback"}]': finished 2024-03-12T03:27:06.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:05 smithi094 ceph-mon[40484]: osdmap e592: 8 total, 8 up, 8 in 2024-03-12T03:27:06.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:05 smithi151 ceph-mon[32923]: pgmap v905: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:27:06.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:05 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:06.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-116", "mode": "writeback"}]': finished 2024-03-12T03:27:06.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:05 smithi151 ceph-mon[32923]: osdmap e592: 8 total, 8 up, 8 in 2024-03-12T03:27:07.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "311.0"}]: dispatch 2024-03-12T03:27:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:06 smithi094 ceph-mon[29291]: from='mon.0 v1:172.21.15.94:0/2786002288' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "311.0"}]: dispatch 2024-03-12T03:27:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:06 smithi094 ceph-mon[29291]: 311.0 scrub starts 2024-03-12T03:27:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:06 smithi094 ceph-mon[29291]: 311.0 scrub ok 2024-03-12T03:27:07.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "311.0"}]: dispatch 2024-03-12T03:27:07.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:06 smithi094 ceph-mon[40484]: from='mon.0 v1:172.21.15.94:0/2786002288' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "311.0"}]: dispatch 2024-03-12T03:27:07.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:06 smithi094 ceph-mon[40484]: 311.0 scrub starts 2024-03-12T03:27:07.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:06 smithi094 ceph-mon[40484]: 311.0 scrub ok 2024-03-12T03:27:07.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "311.0"}]: dispatch 2024-03-12T03:27:07.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:06 smithi151 ceph-mon[32923]: from='mon.0 v1:172.21.15.94:0/2786002288' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "311.0"}]: dispatch 2024-03-12T03:27:07.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:06 smithi151 ceph-mon[32923]: 311.0 scrub starts 2024-03-12T03:27:07.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:06 smithi151 ceph-mon[32923]: 311.0 scrub ok 2024-03-12T03:27:08.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:07 smithi094 ceph-mon[29291]: pgmap v908: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-12T03:27:08.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:07 smithi094 ceph-mon[40484]: pgmap v908: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-12T03:27:08.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:07 smithi151 ceph-mon[32923]: pgmap v908: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-12T03:27:08.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:27:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:27:09.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:08 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:09.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:08 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:27:09.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:08 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:09.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:08 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:27:09.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:08 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:09.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:08 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:27:09.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:09 smithi094 ceph-mon[29291]: pgmap v909: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 521 B/s wr, 2 op/s 2024-03-12T03:27:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:09 smithi094 ceph-mon[40484]: pgmap v909: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 521 B/s wr, 2 op/s 2024-03-12T03:27:10.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:09 smithi151 ceph-mon[32923]: pgmap v909: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 521 B/s wr, 2 op/s 2024-03-12T03:27:10.858 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:27:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:27:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:27:12.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:11 smithi094 ceph-mon[40484]: pgmap v910: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 445 B/s wr, 1 op/s 2024-03-12T03:27:12.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:11 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:12.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:11 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:12.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:11 smithi094 ceph-mon[29291]: pgmap v910: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 445 B/s wr, 1 op/s 2024-03-12T03:27:12.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:11 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:12.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:11 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:12.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:11 smithi151 ceph-mon[32923]: pgmap v910: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 445 B/s wr, 1 op/s 2024-03-12T03:27:12.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:11 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:12.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:11 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:14.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:13 smithi094 ceph-mon[40484]: pgmap v911: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2024-03-12T03:27:14.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:13 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:14.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:13 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:14.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:13 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:27:14.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:13 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:27:14.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:13 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:14.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:13 smithi094 ceph-mon[29291]: pgmap v911: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2024-03-12T03:27:14.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:13 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:14.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:13 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:14.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:13 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:27:14.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:13 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:27:14.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:13 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:14.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:13 smithi151 ceph-mon[32923]: pgmap v911: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2024-03-12T03:27:14.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:13 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:14.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:13 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:14.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:13 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:27:14.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:13 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:27:14.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:13 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:16.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:15 smithi094 ceph-mon[40484]: pgmap v912: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 310 B/s wr, 1 op/s 2024-03-12T03:27:16.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:15 smithi094 ceph-mon[29291]: pgmap v912: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 310 B/s wr, 1 op/s 2024-03-12T03:27:16.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:15 smithi151 ceph-mon[32923]: pgmap v912: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 310 B/s wr, 1 op/s 2024-03-12T03:27:18.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:17 smithi094 ceph-mon[29291]: pgmap v913: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 282 B/s wr, 1 op/s 2024-03-12T03:27:18.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:17 smithi094 ceph-mon[40484]: pgmap v913: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 282 B/s wr, 1 op/s 2024-03-12T03:27:18.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:17 smithi151 ceph-mon[32923]: pgmap v913: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 282 B/s wr, 1 op/s 2024-03-12T03:27:18.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:27:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:27:19.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:18 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:19.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:18 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:19.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:18 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:20.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:27:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:27:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:27:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:20 smithi094 ceph-mon[40484]: pgmap v914: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:20 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:27:20.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:20 smithi094 ceph-mon[29291]: pgmap v914: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:20.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:20 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:20.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:27:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:20 smithi151 ceph-mon[32923]: pgmap v914: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:20 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:27:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:22 smithi094 ceph-mon[29291]: pgmap v915: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:27:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:22 smithi094 ceph-mon[29291]: osdmap e593: 8 total, 8 up, 8 in 2024-03-12T03:27:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:22 smithi094 ceph-mon[40484]: pgmap v915: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:27:22.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:22 smithi094 ceph-mon[40484]: osdmap e593: 8 total, 8 up, 8 in 2024-03-12T03:27:22.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:22 smithi151 ceph-mon[32923]: pgmap v915: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:27:22.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:22 smithi151 ceph-mon[32923]: osdmap e593: 8 total, 8 up, 8 in 2024-03-12T03:27:22.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:23.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:23.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:23 smithi094 ceph-mon[29291]: osdmap e594: 8 total, 8 up, 8 in 2024-03-12T03:27:23.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-116"}]: dispatch 2024-03-12T03:27:23.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:23 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:23.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-116"}]': finished 2024-03-12T03:27:23.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:23 smithi094 ceph-mon[29291]: osdmap e595: 8 total, 8 up, 8 in 2024-03-12T03:27:23.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:23.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:23 smithi094 ceph-mon[40484]: osdmap e594: 8 total, 8 up, 8 in 2024-03-12T03:27:23.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-116"}]: dispatch 2024-03-12T03:27:23.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:23 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:23.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-116"}]': finished 2024-03-12T03:27:23.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:23 smithi094 ceph-mon[40484]: osdmap e595: 8 total, 8 up, 8 in 2024-03-12T03:27:23.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:23.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:23 smithi151 ceph-mon[32923]: osdmap e594: 8 total, 8 up, 8 in 2024-03-12T03:27:23.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-116"}]: dispatch 2024-03-12T03:27:23.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:23 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:23.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-116"}]': finished 2024-03-12T03:27:23.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:23 smithi151 ceph-mon[32923]: osdmap e595: 8 total, 8 up, 8 in 2024-03-12T03:27:25.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:24 smithi094 ceph-mon[29291]: pgmap v918: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:27:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:24 smithi094 ceph-mon[40484]: pgmap v918: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:27:25.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:24 smithi151 ceph-mon[32923]: pgmap v918: 268 pgs: 268 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:27:26.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:25 smithi094 ceph-mon[29291]: osdmap e596: 8 total, 8 up, 8 in 2024-03-12T03:27:26.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:25 smithi094 ceph-mon[29291]: pgmap v921: 236 pgs: 236 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-12T03:27:26.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:25 smithi094 ceph-mon[29291]: osdmap e597: 8 total, 8 up, 8 in 2024-03-12T03:27:26.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:26.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:25 smithi094 ceph-mon[40484]: osdmap e596: 8 total, 8 up, 8 in 2024-03-12T03:27:26.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:25 smithi094 ceph-mon[40484]: pgmap v921: 236 pgs: 236 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-12T03:27:26.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:25 smithi094 ceph-mon[40484]: osdmap e597: 8 total, 8 up, 8 in 2024-03-12T03:27:26.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:26.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:25 smithi151 ceph-mon[32923]: osdmap e596: 8 total, 8 up, 8 in 2024-03-12T03:27:26.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:25 smithi151 ceph-mon[32923]: pgmap v921: 236 pgs: 236 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-12T03:27:26.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:25 smithi151 ceph-mon[32923]: osdmap e597: 8 total, 8 up, 8 in 2024-03-12T03:27:26.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:27 smithi094 ceph-mon[29291]: osdmap e598: 8 total, 8 up, 8 in 2024-03-12T03:27:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:27 smithi094 ceph-mon[40484]: osdmap e598: 8 total, 8 up, 8 in 2024-03-12T03:27:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:27 smithi151 ceph-mon[32923]: osdmap e598: 8 total, 8 up, 8 in 2024-03-12T03:27:28.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:27:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:27:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:28 smithi094 ceph-mon[29291]: pgmap v924: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:27:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:28 smithi094 ceph-mon[29291]: osdmap e599: 8 total, 8 up, 8 in 2024-03-12T03:27:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:28 smithi094 ceph-mon[29291]: osdmap e600: 8 total, 8 up, 8 in 2024-03-12T03:27:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-118"}]: dispatch 2024-03-12T03:27:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:28 smithi094 ceph-mon[40484]: pgmap v924: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:27:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:28 smithi094 ceph-mon[40484]: osdmap e599: 8 total, 8 up, 8 in 2024-03-12T03:27:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:28 smithi094 ceph-mon[40484]: osdmap e600: 8 total, 8 up, 8 in 2024-03-12T03:27:28.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-118"}]: dispatch 2024-03-12T03:27:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:28 smithi151 ceph-mon[32923]: pgmap v924: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:27:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:28 smithi151 ceph-mon[32923]: osdmap e599: 8 total, 8 up, 8 in 2024-03-12T03:27:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:28 smithi151 ceph-mon[32923]: osdmap e600: 8 total, 8 up, 8 in 2024-03-12T03:27:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-118"}]: dispatch 2024-03-12T03:27:29.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-118"}]': finished 2024-03-12T03:27:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:29 smithi094 ceph-mon[29291]: osdmap e601: 8 total, 8 up, 8 in 2024-03-12T03:27:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-118", "mode": "writeback"}]: dispatch 2024-03-12T03:27:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-118"}]': finished 2024-03-12T03:27:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:29 smithi094 ceph-mon[40484]: osdmap e601: 8 total, 8 up, 8 in 2024-03-12T03:27:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-118", "mode": "writeback"}]: dispatch 2024-03-12T03:27:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-118"}]': finished 2024-03-12T03:27:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:29 smithi151 ceph-mon[32923]: osdmap e601: 8 total, 8 up, 8 in 2024-03-12T03:27:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-118", "mode": "writeback"}]: dispatch 2024-03-12T03:27:30.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:30 smithi094 ceph-mon[29291]: pgmap v927: 268 pgs: 268 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:27:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:30 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-118", "mode": "writeback"}]': finished 2024-03-12T03:27:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:30 smithi094 ceph-mon[29291]: osdmap e602: 8 total, 8 up, 8 in 2024-03-12T03:27:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:30 smithi094 ceph-mon[40484]: pgmap v927: 268 pgs: 268 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:27:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:30 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-118", "mode": "writeback"}]': finished 2024-03-12T03:27:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:30 smithi094 ceph-mon[40484]: osdmap e602: 8 total, 8 up, 8 in 2024-03-12T03:27:30.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:27:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:27:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:27:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:30 smithi151 ceph-mon[32923]: pgmap v927: 268 pgs: 268 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:27:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:30 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-118", "mode": "writeback"}]': finished 2024-03-12T03:27:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:30 smithi151 ceph-mon[32923]: osdmap e602: 8 total, 8 up, 8 in 2024-03-12T03:27:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:32 smithi094 ceph-mon[29291]: pgmap v930: 268 pgs: 268 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:27:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:32 smithi094 ceph-mon[29291]: osdmap e603: 8 total, 8 up, 8 in 2024-03-12T03:27:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:32 smithi094 ceph-mon[40484]: pgmap v930: 268 pgs: 268 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:27:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:32 smithi094 ceph-mon[40484]: osdmap e603: 8 total, 8 up, 8 in 2024-03-12T03:27:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:32 smithi151 ceph-mon[32923]: pgmap v930: 268 pgs: 268 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:27:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:32 smithi151 ceph-mon[32923]: osdmap e603: 8 total, 8 up, 8 in 2024-03-12T03:27:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:33 smithi094 ceph-mon[29291]: osdmap e604: 8 total, 8 up, 8 in 2024-03-12T03:27:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-118"}]: dispatch 2024-03-12T03:27:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:33 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-118"}]': finished 2024-03-12T03:27:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:33 smithi094 ceph-mon[29291]: osdmap e605: 8 total, 8 up, 8 in 2024-03-12T03:27:33.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:33 smithi094 ceph-mon[40484]: osdmap e604: 8 total, 8 up, 8 in 2024-03-12T03:27:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-118"}]: dispatch 2024-03-12T03:27:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:33 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-118"}]': finished 2024-03-12T03:27:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:33 smithi094 ceph-mon[40484]: osdmap e605: 8 total, 8 up, 8 in 2024-03-12T03:27:33.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:33.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:33 smithi151 ceph-mon[32923]: osdmap e604: 8 total, 8 up, 8 in 2024-03-12T03:27:33.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-118"}]: dispatch 2024-03-12T03:27:33.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:33 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:33.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-118"}]': finished 2024-03-12T03:27:33.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:33 smithi151 ceph-mon[32923]: osdmap e605: 8 total, 8 up, 8 in 2024-03-12T03:27:34.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:34 smithi094 ceph-mon[29291]: pgmap v933: 268 pgs: 268 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:27:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:34 smithi094 ceph-mon[40484]: pgmap v933: 268 pgs: 268 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:27:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:34 smithi151 ceph-mon[32923]: pgmap v933: 268 pgs: 268 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:27:35.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:35 smithi094 ceph-mon[29291]: osdmap e606: 8 total, 8 up, 8 in 2024-03-12T03:27:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:35 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:27:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:35 smithi094 ceph-mon[40484]: osdmap e606: 8 total, 8 up, 8 in 2024-03-12T03:27:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:35 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:27:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:35 smithi151 ceph-mon[32923]: osdmap e606: 8 total, 8 up, 8 in 2024-03-12T03:27:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:35 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:27:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:27:36.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:36 smithi094 ceph-mon[29291]: pgmap v936: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:27:36.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:36 smithi094 ceph-mon[29291]: osdmap e607: 8 total, 8 up, 8 in 2024-03-12T03:27:36.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:36.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:36.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:36 smithi094 ceph-mon[29291]: osdmap e608: 8 total, 8 up, 8 in 2024-03-12T03:27:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:36 smithi094 ceph-mon[40484]: pgmap v936: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:27:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:36 smithi094 ceph-mon[40484]: osdmap e607: 8 total, 8 up, 8 in 2024-03-12T03:27:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:36.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:36 smithi094 ceph-mon[40484]: osdmap e608: 8 total, 8 up, 8 in 2024-03-12T03:27:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:36 smithi151 ceph-mon[32923]: pgmap v936: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:27:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:36 smithi151 ceph-mon[32923]: osdmap e607: 8 total, 8 up, 8 in 2024-03-12T03:27:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:36 smithi151 ceph-mon[32923]: osdmap e608: 8 total, 8 up, 8 in 2024-03-12T03:27:37.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:37 smithi094 ceph-mon[29291]: osdmap e609: 8 total, 8 up, 8 in 2024-03-12T03:27:37.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-120"}]: dispatch 2024-03-12T03:27:37.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:37.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:37.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:37 smithi094 ceph-mon[40484]: osdmap e609: 8 total, 8 up, 8 in 2024-03-12T03:27:37.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-120"}]: dispatch 2024-03-12T03:27:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:37 smithi151 ceph-mon[32923]: osdmap e609: 8 total, 8 up, 8 in 2024-03-12T03:27:37.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-120"}]: dispatch 2024-03-12T03:27:38.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:38 smithi094 ceph-mon[29291]: pgmap v939: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-12T03:27:38.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-120"}]': finished 2024-03-12T03:27:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:38 smithi094 ceph-mon[29291]: osdmap e610: 8 total, 8 up, 8 in 2024-03-12T03:27:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-120", "mode": "writeback"}]: dispatch 2024-03-12T03:27:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:38 smithi094 ceph-mon[40484]: pgmap v939: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-12T03:27:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-120"}]': finished 2024-03-12T03:27:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:38 smithi094 ceph-mon[40484]: osdmap e610: 8 total, 8 up, 8 in 2024-03-12T03:27:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-120", "mode": "writeback"}]: dispatch 2024-03-12T03:27:38.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:27:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:27:38.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:38 smithi151 ceph-mon[32923]: pgmap v939: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-12T03:27:38.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-120"}]': finished 2024-03-12T03:27:38.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:38 smithi151 ceph-mon[32923]: osdmap e610: 8 total, 8 up, 8 in 2024-03-12T03:27:38.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-120", "mode": "writeback"}]: dispatch 2024-03-12T03:27:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:39 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-120", "mode": "writeback"}]': finished 2024-03-12T03:27:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:39 smithi094 ceph-mon[29291]: osdmap e611: 8 total, 8 up, 8 in 2024-03-12T03:27:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:39 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-120", "mode": "writeback"}]': finished 2024-03-12T03:27:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:39 smithi094 ceph-mon[40484]: osdmap e611: 8 total, 8 up, 8 in 2024-03-12T03:27:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:39 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-120", "mode": "writeback"}]': finished 2024-03-12T03:27:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:39 smithi151 ceph-mon[32923]: osdmap e611: 8 total, 8 up, 8 in 2024-03-12T03:27:40.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:40 smithi094 ceph-mon[29291]: pgmap v942: 268 pgs: 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:40.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:40.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:27:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:27:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:27:40.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:40 smithi094 ceph-mon[40484]: pgmap v942: 268 pgs: 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:40.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:40 smithi151 ceph-mon[32923]: pgmap v942: 268 pgs: 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:41 smithi094 ceph-mon[29291]: osdmap e612: 8 total, 8 up, 8 in 2024-03-12T03:27:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-120"}]: dispatch 2024-03-12T03:27:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:41 smithi094 ceph-mon[40484]: osdmap e612: 8 total, 8 up, 8 in 2024-03-12T03:27:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-120"}]: dispatch 2024-03-12T03:27:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:41 smithi151 ceph-mon[32923]: osdmap e612: 8 total, 8 up, 8 in 2024-03-12T03:27:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-120"}]: dispatch 2024-03-12T03:27:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:42 smithi094 ceph-mon[29291]: pgmap v945: 268 pgs: 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:42 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-120"}]': finished 2024-03-12T03:27:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:42 smithi094 ceph-mon[29291]: osdmap e613: 8 total, 8 up, 8 in 2024-03-12T03:27:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:42 smithi094 ceph-mon[40484]: pgmap v945: 268 pgs: 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:42 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-120"}]': finished 2024-03-12T03:27:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:42 smithi094 ceph-mon[40484]: osdmap e613: 8 total, 8 up, 8 in 2024-03-12T03:27:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:42 smithi151 ceph-mon[32923]: pgmap v945: 268 pgs: 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:42 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-120"}]': finished 2024-03-12T03:27:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:42 smithi151 ceph-mon[32923]: osdmap e613: 8 total, 8 up, 8 in 2024-03-12T03:27:43.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:43 smithi094 ceph-mon[29291]: osdmap e614: 8 total, 8 up, 8 in 2024-03-12T03:27:43.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:43 smithi094 ceph-mon[29291]: osdmap e615: 8 total, 8 up, 8 in 2024-03-12T03:27:43.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:43 smithi094 ceph-mon[40484]: osdmap e614: 8 total, 8 up, 8 in 2024-03-12T03:27:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:43 smithi094 ceph-mon[40484]: osdmap e615: 8 total, 8 up, 8 in 2024-03-12T03:27:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:43 smithi151 ceph-mon[32923]: osdmap e614: 8 total, 8 up, 8 in 2024-03-12T03:27:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:43 smithi151 ceph-mon[32923]: osdmap e615: 8 total, 8 up, 8 in 2024-03-12T03:27:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:44.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:44 smithi151 ceph-mon[32923]: pgmap v948: 236 pgs: 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:44.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:44.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:44 smithi151 ceph-mon[32923]: osdmap e616: 8 total, 8 up, 8 in 2024-03-12T03:27:44.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:44 smithi094 ceph-mon[29291]: pgmap v948: 236 pgs: 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:44 smithi094 ceph-mon[29291]: osdmap e616: 8 total, 8 up, 8 in 2024-03-12T03:27:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:44.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:44 smithi094 ceph-mon[40484]: pgmap v948: 236 pgs: 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:44.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:44.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:44 smithi094 ceph-mon[40484]: osdmap e616: 8 total, 8 up, 8 in 2024-03-12T03:27:44.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:46.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:46 smithi094 ceph-mon[29291]: pgmap v951: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:46.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:46.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:46 smithi094 ceph-mon[29291]: osdmap e617: 8 total, 8 up, 8 in 2024-03-12T03:27:46.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-122"}]: dispatch 2024-03-12T03:27:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:46 smithi094 ceph-mon[40484]: pgmap v951: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:46 smithi094 ceph-mon[40484]: osdmap e617: 8 total, 8 up, 8 in 2024-03-12T03:27:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-122"}]: dispatch 2024-03-12T03:27:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:46 smithi151 ceph-mon[32923]: pgmap v951: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:27:46.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:46.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:46 smithi151 ceph-mon[32923]: osdmap e617: 8 total, 8 up, 8 in 2024-03-12T03:27:46.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-122"}]: dispatch 2024-03-12T03:27:47.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-122"}]': finished 2024-03-12T03:27:47.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:47 smithi094 ceph-mon[40484]: osdmap e618: 8 total, 8 up, 8 in 2024-03-12T03:27:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-122", "mode": "writeback"}]: dispatch 2024-03-12T03:27:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:47 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-122", "mode": "writeback"}]': finished 2024-03-12T03:27:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:47 smithi094 ceph-mon[40484]: osdmap e619: 8 total, 8 up, 8 in 2024-03-12T03:27:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-122"}]': finished 2024-03-12T03:27:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:47 smithi094 ceph-mon[29291]: osdmap e618: 8 total, 8 up, 8 in 2024-03-12T03:27:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-122", "mode": "writeback"}]: dispatch 2024-03-12T03:27:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:47 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-122", "mode": "writeback"}]': finished 2024-03-12T03:27:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:47 smithi094 ceph-mon[29291]: osdmap e619: 8 total, 8 up, 8 in 2024-03-12T03:27:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-122"}]': finished 2024-03-12T03:27:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:47 smithi151 ceph-mon[32923]: osdmap e618: 8 total, 8 up, 8 in 2024-03-12T03:27:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-122", "mode": "writeback"}]: dispatch 2024-03-12T03:27:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:47 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-122", "mode": "writeback"}]': finished 2024-03-12T03:27:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:47 smithi151 ceph-mon[32923]: osdmap e619: 8 total, 8 up, 8 in 2024-03-12T03:27:48.611 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:48 smithi094 ceph-mon[29291]: pgmap v954: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:27:48.611 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:48.611 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:48 smithi094 ceph-mon[40484]: pgmap v954: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:27:48.611 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:48 smithi151 ceph-mon[32923]: pgmap v954: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:27:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:48.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:27:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:27:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:49 smithi094 ceph-mon[40484]: osdmap e620: 8 total, 8 up, 8 in 2024-03-12T03:27:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:49 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-122"}]: dispatch 2024-03-12T03:27:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:49 smithi094 ceph-mon[29291]: osdmap e620: 8 total, 8 up, 8 in 2024-03-12T03:27:49.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:49 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-122"}]: dispatch 2024-03-12T03:27:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:49 smithi151 ceph-mon[32923]: osdmap e620: 8 total, 8 up, 8 in 2024-03-12T03:27:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:49 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-122"}]: dispatch 2024-03-12T03:27:50.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:50 smithi094 ceph-mon[29291]: pgmap v957: 268 pgs: 268 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-12T03:27:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:50 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-122"}]': finished 2024-03-12T03:27:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:50 smithi094 ceph-mon[29291]: osdmap e621: 8 total, 8 up, 8 in 2024-03-12T03:27:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:27:50.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:27:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:27:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:27:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:50 smithi094 ceph-mon[40484]: pgmap v957: 268 pgs: 268 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-12T03:27:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:50 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-122"}]': finished 2024-03-12T03:27:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:50 smithi094 ceph-mon[40484]: osdmap e621: 8 total, 8 up, 8 in 2024-03-12T03:27:50.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:27:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:50 smithi151 ceph-mon[32923]: pgmap v957: 268 pgs: 268 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-12T03:27:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:50 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-122"}]': finished 2024-03-12T03:27:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:50 smithi151 ceph-mon[32923]: osdmap e621: 8 total, 8 up, 8 in 2024-03-12T03:27:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:27:51.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:51 smithi094 ceph-mon[29291]: osdmap e622: 8 total, 8 up, 8 in 2024-03-12T03:27:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:51 smithi094 ceph-mon[40484]: osdmap e622: 8 total, 8 up, 8 in 2024-03-12T03:27:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:51 smithi151 ceph-mon[32923]: osdmap e622: 8 total, 8 up, 8 in 2024-03-12T03:27:52.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:52 smithi094 ceph-mon[29291]: pgmap v960: 236 pgs: 236 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:27:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:52 smithi094 ceph-mon[29291]: osdmap e623: 8 total, 8 up, 8 in 2024-03-12T03:27:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:52 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:52 smithi094 ceph-mon[40484]: pgmap v960: 236 pgs: 236 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:27:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:52 smithi094 ceph-mon[40484]: osdmap e623: 8 total, 8 up, 8 in 2024-03-12T03:27:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:52 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:52 smithi151 ceph-mon[32923]: pgmap v960: 236 pgs: 236 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T03:27:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:52 smithi151 ceph-mon[32923]: osdmap e623: 8 total, 8 up, 8 in 2024-03-12T03:27:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:52 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:27:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:53 smithi094 ceph-mon[40484]: osdmap e624: 8 total, 8 up, 8 in 2024-03-12T03:27:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:53 smithi094 ceph-mon[40484]: osdmap e625: 8 total, 8 up, 8 in 2024-03-12T03:27:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:53 smithi094 ceph-mon[29291]: osdmap e624: 8 total, 8 up, 8 in 2024-03-12T03:27:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:53 smithi094 ceph-mon[29291]: osdmap e625: 8 total, 8 up, 8 in 2024-03-12T03:27:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:27:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:53 smithi151 ceph-mon[32923]: osdmap e624: 8 total, 8 up, 8 in 2024-03-12T03:27:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:53 smithi151 ceph-mon[32923]: osdmap e625: 8 total, 8 up, 8 in 2024-03-12T03:27:54.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:54 smithi094 ceph-mon[40484]: pgmap v963: 268 pgs: 16 unknown, 252 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:27:54.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:54.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:54.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:54 smithi094 ceph-mon[40484]: osdmap e626: 8 total, 8 up, 8 in 2024-03-12T03:27:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:54 smithi094 ceph-mon[29291]: pgmap v963: 268 pgs: 16 unknown, 252 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:27:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:54 smithi094 ceph-mon[29291]: osdmap e626: 8 total, 8 up, 8 in 2024-03-12T03:27:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:54 smithi151 ceph-mon[32923]: pgmap v963: 268 pgs: 16 unknown, 252 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:27:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:27:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:27:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:54 smithi151 ceph-mon[32923]: osdmap e626: 8 total, 8 up, 8 in 2024-03-12T03:27:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-124"}]: dispatch 2024-03-12T03:27:55.753 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-124"}]': finished 2024-03-12T03:27:55.753 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:55 smithi094 ceph-mon[29291]: osdmap e627: 8 total, 8 up, 8 in 2024-03-12T03:27:55.753 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-124", "mode": "writeback"}]: dispatch 2024-03-12T03:27:55.753 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-124"}]: dispatch 2024-03-12T03:27:55.753 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-124"}]': finished 2024-03-12T03:27:55.754 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:55 smithi094 ceph-mon[40484]: osdmap e627: 8 total, 8 up, 8 in 2024-03-12T03:27:55.754 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-124", "mode": "writeback"}]: dispatch 2024-03-12T03:27:55.754 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-124"}]: dispatch 2024-03-12T03:27:55.754 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-124"}]': finished 2024-03-12T03:27:55.754 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:55 smithi151 ceph-mon[32923]: osdmap e627: 8 total, 8 up, 8 in 2024-03-12T03:27:55.755 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-124", "mode": "writeback"}]: dispatch 2024-03-12T03:27:56.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:56 smithi094 ceph-mon[40484]: pgmap v966: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:27:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:56 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-124", "mode": "writeback"}]': finished 2024-03-12T03:27:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:56 smithi094 ceph-mon[40484]: osdmap e628: 8 total, 8 up, 8 in 2024-03-12T03:27:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:56 smithi094 ceph-mon[29291]: pgmap v966: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:27:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:56 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-124", "mode": "writeback"}]': finished 2024-03-12T03:27:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:56 smithi094 ceph-mon[29291]: osdmap e628: 8 total, 8 up, 8 in 2024-03-12T03:27:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:56 smithi151 ceph-mon[32923]: pgmap v966: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:27:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:56 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:27:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-124", "mode": "writeback"}]': finished 2024-03-12T03:27:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:56 smithi151 ceph-mon[32923]: osdmap e628: 8 total, 8 up, 8 in 2024-03-12T03:27:58.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:58 smithi094 ceph-mon[40484]: pgmap v969: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-12T03:27:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:58 smithi094 ceph-mon[40484]: osdmap e629: 8 total, 8 up, 8 in 2024-03-12T03:27:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:58 smithi094 ceph-mon[29291]: pgmap v969: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-12T03:27:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:58 smithi094 ceph-mon[29291]: osdmap e629: 8 total, 8 up, 8 in 2024-03-12T03:27:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:58.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:27:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:27:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:58 smithi151 ceph-mon[32923]: pgmap v969: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-12T03:27:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:58 smithi151 ceph-mon[32923]: osdmap e629: 8 total, 8 up, 8 in 2024-03-12T03:27:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:27:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:59 smithi094 ceph-mon[40484]: osdmap e630: 8 total, 8 up, 8 in 2024-03-12T03:27:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-124"}]: dispatch 2024-03-12T03:27:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:59 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:27:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-124"}]': finished 2024-03-12T03:27:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:59.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:59 smithi094 ceph-mon[29291]: osdmap e630: 8 total, 8 up, 8 in 2024-03-12T03:27:59.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-124"}]: dispatch 2024-03-12T03:27:59.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:59 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:59.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:27:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-124"}]': finished 2024-03-12T03:27:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:27:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:27:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:59 smithi151 ceph-mon[32923]: osdmap e630: 8 total, 8 up, 8 in 2024-03-12T03:27:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-124"}]: dispatch 2024-03-12T03:27:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:59 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:27:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:27:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-124"}]': finished 2024-03-12T03:28:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:00 smithi094 ceph-mon[29291]: pgmap v972: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-12T03:28:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:00 smithi094 ceph-mon[29291]: osdmap e631: 8 total, 8 up, 8 in 2024-03-12T03:28:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:00 smithi094 ceph-mon[40484]: pgmap v972: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-12T03:28:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:00 smithi094 ceph-mon[40484]: osdmap e631: 8 total, 8 up, 8 in 2024-03-12T03:28:00.711 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:28:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:28:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:28:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:00 smithi151 ceph-mon[32923]: pgmap v972: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-12T03:28:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:00 smithi151 ceph-mon[32923]: osdmap e631: 8 total, 8 up, 8 in 2024-03-12T03:28:01.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:01 smithi094 ceph-mon[29291]: osdmap e632: 8 total, 8 up, 8 in 2024-03-12T03:28:01.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:01 smithi094 ceph-mon[40484]: osdmap e632: 8 total, 8 up, 8 in 2024-03-12T03:28:01.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:01 smithi151 ceph-mon[32923]: osdmap e632: 8 total, 8 up, 8 in 2024-03-12T03:28:02.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:02 smithi094 ceph-mon[29291]: pgmap v975: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:28:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:02 smithi094 ceph-mon[29291]: osdmap e633: 8 total, 8 up, 8 in 2024-03-12T03:28:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:02 smithi094 ceph-mon[40484]: pgmap v975: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:28:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:02 smithi094 ceph-mon[40484]: osdmap e633: 8 total, 8 up, 8 in 2024-03-12T03:28:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:02 smithi151 ceph-mon[32923]: pgmap v975: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:28:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:02 smithi151 ceph-mon[32923]: osdmap e633: 8 total, 8 up, 8 in 2024-03-12T03:28:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:03 smithi094 ceph-mon[29291]: osdmap e634: 8 total, 8 up, 8 in 2024-03-12T03:28:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:03 smithi094 ceph-mon[29291]: osdmap e635: 8 total, 8 up, 8 in 2024-03-12T03:28:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-126"}]: dispatch 2024-03-12T03:28:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:03.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:03 smithi094 ceph-mon[40484]: osdmap e634: 8 total, 8 up, 8 in 2024-03-12T03:28:03.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:03.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:03.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:03 smithi094 ceph-mon[40484]: osdmap e635: 8 total, 8 up, 8 in 2024-03-12T03:28:03.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-126"}]: dispatch 2024-03-12T03:28:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:03 smithi151 ceph-mon[32923]: osdmap e634: 8 total, 8 up, 8 in 2024-03-12T03:28:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:03.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:03.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:03 smithi151 ceph-mon[32923]: osdmap e635: 8 total, 8 up, 8 in 2024-03-12T03:28:03.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-126"}]: dispatch 2024-03-12T03:28:04.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:04 smithi094 ceph-mon[29291]: pgmap v978: 268 pgs: 14 unknown, 254 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-126"}]': finished 2024-03-12T03:28:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:04 smithi094 ceph-mon[29291]: osdmap e636: 8 total, 8 up, 8 in 2024-03-12T03:28:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-126", "mode": "writeback"}]: dispatch 2024-03-12T03:28:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:04 smithi094 ceph-mon[40484]: pgmap v978: 268 pgs: 14 unknown, 254 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-126"}]': finished 2024-03-12T03:28:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:04 smithi094 ceph-mon[40484]: osdmap e636: 8 total, 8 up, 8 in 2024-03-12T03:28:04.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-126", "mode": "writeback"}]: dispatch 2024-03-12T03:28:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:04 smithi151 ceph-mon[32923]: pgmap v978: 268 pgs: 14 unknown, 254 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-126"}]': finished 2024-03-12T03:28:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:04 smithi151 ceph-mon[32923]: osdmap e636: 8 total, 8 up, 8 in 2024-03-12T03:28:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-126", "mode": "writeback"}]: dispatch 2024-03-12T03:28:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:28:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:05 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-126", "mode": "writeback"}]': finished 2024-03-12T03:28:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:05 smithi151 ceph-mon[32923]: osdmap e637: 8 total, 8 up, 8 in 2024-03-12T03:28:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:28:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:05 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-126", "mode": "writeback"}]': finished 2024-03-12T03:28:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:05 smithi094 ceph-mon[29291]: osdmap e637: 8 total, 8 up, 8 in 2024-03-12T03:28:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:28:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:05 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-126", "mode": "writeback"}]': finished 2024-03-12T03:28:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:05 smithi094 ceph-mon[40484]: osdmap e637: 8 total, 8 up, 8 in 2024-03-12T03:28:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:06 smithi151 ceph-mon[32923]: pgmap v981: 268 pgs: 268 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:06.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:06 smithi151 ceph-mon[32923]: osdmap e638: 8 total, 8 up, 8 in 2024-03-12T03:28:06.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-126"}]: dispatch 2024-03-12T03:28:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:06 smithi094 ceph-mon[29291]: pgmap v981: 268 pgs: 268 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:06 smithi094 ceph-mon[29291]: osdmap e638: 8 total, 8 up, 8 in 2024-03-12T03:28:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-126"}]: dispatch 2024-03-12T03:28:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:06 smithi094 ceph-mon[40484]: pgmap v981: 268 pgs: 268 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:06 smithi094 ceph-mon[40484]: osdmap e638: 8 total, 8 up, 8 in 2024-03-12T03:28:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-126"}]: dispatch 2024-03-12T03:28:07.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:07 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-126"}]': finished 2024-03-12T03:28:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:07 smithi094 ceph-mon[29291]: osdmap e639: 8 total, 8 up, 8 in 2024-03-12T03:28:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:07 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-126"}]': finished 2024-03-12T03:28:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:07 smithi094 ceph-mon[40484]: osdmap e639: 8 total, 8 up, 8 in 2024-03-12T03:28:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:07 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-126"}]': finished 2024-03-12T03:28:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:07 smithi151 ceph-mon[32923]: osdmap e639: 8 total, 8 up, 8 in 2024-03-12T03:28:08.277 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:28:08.277 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9041 ms) 2024-03-12T03:28:08.277 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-12T03:28:08.277 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:28:08.277 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6041 ms) 2024-03-12T03:28:08.277 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-12T03:28:08.277 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:28:08.277 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5035 ms) 2024-03-12T03:28:08.277 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-12T03:28:08.277 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:28:08.277 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25423 ms) 2024-03-12T03:28:08.278 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-12T03:28:08.278 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:28:08.278 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14056 ms) 2024-03-12T03:28:08.278 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-12T03:28:08.278 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12091 ms) 2024-03-12T03:28:08.278 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-12T03:28:08.278 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-12T03:28:08.278 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-12T03:28:08.278 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: 00000012 2024-03-12T03:28:08.278 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12066 ms) 2024-03-12T03:28:08.278 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-12T03:28:08.278 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: require_osd_release = squid 2024-03-12T03:28:08.278 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6045 ms) 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (557461 ms total) 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1021 ms) 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11065 ms) 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18107 ms) 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30195 ms total) 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: 2024-03-12T03:28:08.279 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-12T03:28:08.280 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-12T03:28:08.280 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7047 ms) 2024-03-12T03:28:08.280 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-12T03:28:08.280 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8034 ms) 2024-03-12T03:28:08.280 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-12T03:28:08.280 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: waiting for scrub... 2024-03-12T03:28:08.280 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: done waiting 2024-03-12T03:28:08.280 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24684 ms) 2024-03-12T03:28:08.280 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-12T03:28:08.280 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9750 ms) 2024-03-12T03:28:08.280 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-12T03:28:08.280 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8039 ms) 2024-03-12T03:28:08.280 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-12T03:28:08.281 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8038 ms) 2024-03-12T03:28:08.281 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-12T03:28:08.281 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10063 ms) 2024-03-12T03:28:08.281 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-12T03:28:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:08 smithi151 ceph-mon[32923]: pgmap v984: 268 pgs: 268 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:08 smithi151 ceph-mon[32923]: osdmap e640: 8 total, 8 up, 8 in 2024-03-12T03:28:08.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:28:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:28:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:08 smithi094 ceph-mon[29291]: pgmap v984: 268 pgs: 268 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:08 smithi094 ceph-mon[29291]: osdmap e640: 8 total, 8 up, 8 in 2024-03-12T03:28:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:08 smithi094 ceph-mon[40484]: pgmap v984: 268 pgs: 268 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:08 smithi094 ceph-mon[40484]: osdmap e640: 8 total, 8 up, 8 in 2024-03-12T03:28:09.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:09 smithi094 ceph-mon[29291]: osdmap e641: 8 total, 8 up, 8 in 2024-03-12T03:28:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:09 smithi094 ceph-mon[40484]: osdmap e641: 8 total, 8 up, 8 in 2024-03-12T03:28:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:09 smithi151 ceph-mon[32923]: osdmap e641: 8 total, 8 up, 8 in 2024-03-12T03:28:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:10.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:28:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:28:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:28:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:10 smithi094 ceph-mon[29291]: pgmap v987: 236 pgs: 236 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:28:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:10 smithi094 ceph-mon[29291]: osdmap e642: 8 total, 8 up, 8 in 2024-03-12T03:28:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:10 smithi094 ceph-mon[40484]: pgmap v987: 236 pgs: 236 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:28:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:10.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:10 smithi094 ceph-mon[40484]: osdmap e642: 8 total, 8 up, 8 in 2024-03-12T03:28:10.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:10 smithi151 ceph-mon[32923]: pgmap v987: 236 pgs: 236 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:28:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:10 smithi151 ceph-mon[32923]: osdmap e642: 8 total, 8 up, 8 in 2024-03-12T03:28:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:12 smithi094 ceph-mon[29291]: pgmap v990: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:28:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:12 smithi094 ceph-mon[29291]: osdmap e643: 8 total, 8 up, 8 in 2024-03-12T03:28:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-128"}]: dispatch 2024-03-12T03:28:12.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:12 smithi094 ceph-mon[40484]: pgmap v990: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:28:12.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:12.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:12 smithi094 ceph-mon[40484]: osdmap e643: 8 total, 8 up, 8 in 2024-03-12T03:28:12.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-128"}]: dispatch 2024-03-12T03:28:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:12 smithi151 ceph-mon[32923]: pgmap v990: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:28:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:12 smithi151 ceph-mon[32923]: osdmap e643: 8 total, 8 up, 8 in 2024-03-12T03:28:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-128"}]: dispatch 2024-03-12T03:28:13.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-128"}]': finished 2024-03-12T03:28:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:13 smithi094 ceph-mon[29291]: osdmap e644: 8 total, 8 up, 8 in 2024-03-12T03:28:13.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:13 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-128", "mode": "writeback"}]: dispatch 2024-03-12T03:28:13.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-128"}]': finished 2024-03-12T03:28:13.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:13 smithi094 ceph-mon[40484]: osdmap e644: 8 total, 8 up, 8 in 2024-03-12T03:28:13.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:13 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-128", "mode": "writeback"}]: dispatch 2024-03-12T03:28:13.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-128"}]': finished 2024-03-12T03:28:13.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:13 smithi151 ceph-mon[32923]: osdmap e644: 8 total, 8 up, 8 in 2024-03-12T03:28:13.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:13 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-128", "mode": "writeback"}]: dispatch 2024-03-12T03:28:14.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:14 smithi151 ceph-mon[32923]: pgmap v993: 268 pgs: 15 unknown, 253 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:14.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:14 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:14.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-128", "mode": "writeback"}]': finished 2024-03-12T03:28:14.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:14 smithi151 ceph-mon[32923]: osdmap e645: 8 total, 8 up, 8 in 2024-03-12T03:28:14.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:14 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:28:14.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:14 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:28:14.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:14 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:28:14.616 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:14 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:28:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[29291]: pgmap v993: 268 pgs: 15 unknown, 253 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-128", "mode": "writeback"}]': finished 2024-03-12T03:28:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[29291]: osdmap e645: 8 total, 8 up, 8 in 2024-03-12T03:28:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:28:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:28:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:28:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:28:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[40484]: pgmap v993: 268 pgs: 15 unknown, 253 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-128", "mode": "writeback"}]': finished 2024-03-12T03:28:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[40484]: osdmap e645: 8 total, 8 up, 8 in 2024-03-12T03:28:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:28:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:28:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:28:14.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:14 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:28:16.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:16 smithi094 ceph-mon[29291]: pgmap v995: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:28:16.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:16 smithi094 ceph-mon[40484]: pgmap v995: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:28:16.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:16 smithi151 ceph-mon[32923]: pgmap v995: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:28:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[29291]: pgmap v996: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:28:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.7", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.12", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.7", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.14", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.17", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.12", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.14", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.17", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[40484]: pgmap v996: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:28:18.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.7", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.12", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.7", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.14", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.17", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.12", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.14", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:18 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.17", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:28:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:28:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:18 smithi151 ceph-mon[32923]: pgmap v996: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:28:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:18 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.7", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:18 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.12", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:18 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.7", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:18 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.14", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:18 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.17", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:18 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.12", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:18 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.14", "id": [1, 3]}]: dispatch 2024-03-12T03:28:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:18 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.17", "id": [1, 3]}]: dispatch 2024-03-12T03:28:19.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:19 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.7", "id": [1, 3]}]': finished 2024-03-12T03:28:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:19 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.12", "id": [1, 3]}]': finished 2024-03-12T03:28:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:19 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.14", "id": [1, 3]}]': finished 2024-03-12T03:28:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:19 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.17", "id": [1, 3]}]': finished 2024-03-12T03:28:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:19 smithi094 ceph-mon[29291]: osdmap e646: 8 total, 8 up, 8 in 2024-03-12T03:28:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:19 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.7", "id": [1, 3]}]': finished 2024-03-12T03:28:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:19 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.12", "id": [1, 3]}]': finished 2024-03-12T03:28:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:19 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.14", "id": [1, 3]}]': finished 2024-03-12T03:28:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:19 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.17", "id": [1, 3]}]': finished 2024-03-12T03:28:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:19 smithi094 ceph-mon[40484]: osdmap e646: 8 total, 8 up, 8 in 2024-03-12T03:28:19.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:19 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.7", "id": [1, 3]}]': finished 2024-03-12T03:28:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:19 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.12", "id": [1, 3]}]': finished 2024-03-12T03:28:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:19 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.14", "id": [1, 3]}]': finished 2024-03-12T03:28:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:19 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "317.17", "id": [1, 3]}]': finished 2024-03-12T03:28:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:19 smithi151 ceph-mon[32923]: osdmap e646: 8 total, 8 up, 8 in 2024-03-12T03:28:19.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:20.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:28:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:28:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:28:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:20 smithi094 ceph-mon[29291]: pgmap v998: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 315 B/s wr, 2 op/s 2024-03-12T03:28:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:20 smithi094 ceph-mon[29291]: osdmap e647: 8 total, 8 up, 8 in 2024-03-12T03:28:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:20 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:28:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:28:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:20 smithi094 ceph-mon[40484]: pgmap v998: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 315 B/s wr, 2 op/s 2024-03-12T03:28:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:20 smithi094 ceph-mon[40484]: osdmap e647: 8 total, 8 up, 8 in 2024-03-12T03:28:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:20 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:28:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:28:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:20 smithi151 ceph-mon[32923]: pgmap v998: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 315 B/s wr, 2 op/s 2024-03-12T03:28:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:20 smithi151 ceph-mon[32923]: osdmap e647: 8 total, 8 up, 8 in 2024-03-12T03:28:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:20 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:28:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:28:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:21.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:21 smithi094 ceph-mon[29291]: osdmap e648: 8 total, 8 up, 8 in 2024-03-12T03:28:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-128"}]: dispatch 2024-03-12T03:28:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:21 smithi094 ceph-mon[40484]: osdmap e648: 8 total, 8 up, 8 in 2024-03-12T03:28:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-128"}]: dispatch 2024-03-12T03:28:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:21 smithi151 ceph-mon[32923]: osdmap e648: 8 total, 8 up, 8 in 2024-03-12T03:28:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-128"}]: dispatch 2024-03-12T03:28:23.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:22 smithi094 ceph-mon[29291]: pgmap v1001: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-12T03:28:23.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:22 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:23.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-128"}]': finished 2024-03-12T03:28:23.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:22 smithi094 ceph-mon[29291]: osdmap e649: 8 total, 8 up, 8 in 2024-03-12T03:28:23.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:22 smithi094 ceph-mon[40484]: pgmap v1001: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-12T03:28:23.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:22 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:23.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-128"}]': finished 2024-03-12T03:28:23.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:22 smithi094 ceph-mon[40484]: osdmap e649: 8 total, 8 up, 8 in 2024-03-12T03:28:23.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:22 smithi151 ceph-mon[32923]: pgmap v1001: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-12T03:28:23.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:22 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:23.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-128"}]': finished 2024-03-12T03:28:23.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:22 smithi151 ceph-mon[32923]: osdmap e649: 8 total, 8 up, 8 in 2024-03-12T03:28:24.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:23 smithi094 ceph-mon[29291]: osdmap e650: 8 total, 8 up, 8 in 2024-03-12T03:28:24.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:23 smithi094 ceph-mon[29291]: pgmap v1004: 236 pgs: 236 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:28:24.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:23 smithi094 ceph-mon[29291]: osdmap e651: 8 total, 8 up, 8 in 2024-03-12T03:28:24.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:24.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:23 smithi094 ceph-mon[40484]: osdmap e650: 8 total, 8 up, 8 in 2024-03-12T03:28:24.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:23 smithi094 ceph-mon[40484]: pgmap v1004: 236 pgs: 236 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:28:24.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:23 smithi094 ceph-mon[40484]: osdmap e651: 8 total, 8 up, 8 in 2024-03-12T03:28:24.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:24.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:23 smithi151 ceph-mon[32923]: osdmap e650: 8 total, 8 up, 8 in 2024-03-12T03:28:24.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:23 smithi151 ceph-mon[32923]: pgmap v1004: 236 pgs: 236 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:28:24.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:23 smithi151 ceph-mon[32923]: osdmap e651: 8 total, 8 up, 8 in 2024-03-12T03:28:24.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:25.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:25.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:25 smithi094 ceph-mon[29291]: osdmap e652: 8 total, 8 up, 8 in 2024-03-12T03:28:25.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:25.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:25.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:25 smithi094 ceph-mon[40484]: osdmap e652: 8 total, 8 up, 8 in 2024-03-12T03:28:25.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:25.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:25.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:25 smithi151 ceph-mon[32923]: osdmap e652: 8 total, 8 up, 8 in 2024-03-12T03:28:25.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:26.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:26 smithi094 ceph-mon[29291]: pgmap v1007: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:28:26.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:26.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:26 smithi094 ceph-mon[29291]: osdmap e653: 8 total, 8 up, 8 in 2024-03-12T03:28:26.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-130"}]: dispatch 2024-03-12T03:28:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:26 smithi094 ceph-mon[40484]: pgmap v1007: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:28:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:26 smithi094 ceph-mon[40484]: osdmap e653: 8 total, 8 up, 8 in 2024-03-12T03:28:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-130"}]: dispatch 2024-03-12T03:28:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:26 smithi151 ceph-mon[32923]: pgmap v1007: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:28:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:26 smithi151 ceph-mon[32923]: osdmap e653: 8 total, 8 up, 8 in 2024-03-12T03:28:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-130"}]: dispatch 2024-03-12T03:28:27.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-130"}]': finished 2024-03-12T03:28:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:27 smithi094 ceph-mon[29291]: osdmap e654: 8 total, 8 up, 8 in 2024-03-12T03:28:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-130", "mode": "writeback"}]: dispatch 2024-03-12T03:28:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:27 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:27.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-130", "mode": "writeback"}]': finished 2024-03-12T03:28:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-130"}]': finished 2024-03-12T03:28:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:27 smithi094 ceph-mon[40484]: osdmap e654: 8 total, 8 up, 8 in 2024-03-12T03:28:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-130", "mode": "writeback"}]: dispatch 2024-03-12T03:28:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:27 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:27.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-130", "mode": "writeback"}]': finished 2024-03-12T03:28:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-130"}]': finished 2024-03-12T03:28:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:27 smithi151 ceph-mon[32923]: osdmap e654: 8 total, 8 up, 8 in 2024-03-12T03:28:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-130", "mode": "writeback"}]: dispatch 2024-03-12T03:28:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:27 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:27.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-130", "mode": "writeback"}]': finished 2024-03-12T03:28:28.607 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:28:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:28:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:28 smithi094 ceph-mon[29291]: pgmap v1010: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:28 smithi094 ceph-mon[29291]: osdmap e655: 8 total, 8 up, 8 in 2024-03-12T03:28:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:28 smithi094 ceph-mon[40484]: pgmap v1010: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:28 smithi094 ceph-mon[40484]: osdmap e655: 8 total, 8 up, 8 in 2024-03-12T03:28:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:28 smithi151 ceph-mon[32923]: pgmap v1010: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:28 smithi151 ceph-mon[32923]: osdmap e655: 8 total, 8 up, 8 in 2024-03-12T03:28:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:29 smithi094 ceph-mon[29291]: osdmap e656: 8 total, 8 up, 8 in 2024-03-12T03:28:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-130"}]: dispatch 2024-03-12T03:28:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:29.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:29 smithi094 ceph-mon[40484]: osdmap e656: 8 total, 8 up, 8 in 2024-03-12T03:28:29.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-130"}]: dispatch 2024-03-12T03:28:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:29 smithi151 ceph-mon[32923]: osdmap e656: 8 total, 8 up, 8 in 2024-03-12T03:28:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-130"}]: dispatch 2024-03-12T03:28:30.869 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:28:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:28:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:28:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:30 smithi094 ceph-mon[29291]: pgmap v1013: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-12T03:28:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:30 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-130"}]': finished 2024-03-12T03:28:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:30 smithi094 ceph-mon[29291]: osdmap e657: 8 total, 8 up, 8 in 2024-03-12T03:28:31.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:30 smithi094 ceph-mon[40484]: pgmap v1013: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-12T03:28:31.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:30 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:31.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-130"}]': finished 2024-03-12T03:28:31.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:30 smithi094 ceph-mon[40484]: osdmap e657: 8 total, 8 up, 8 in 2024-03-12T03:28:31.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:30 smithi151 ceph-mon[32923]: pgmap v1013: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-12T03:28:31.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:30 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:31.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-130"}]': finished 2024-03-12T03:28:31.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:30 smithi151 ceph-mon[32923]: osdmap e657: 8 total, 8 up, 8 in 2024-03-12T03:28:32.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:31 smithi094 ceph-mon[29291]: osdmap e658: 8 total, 8 up, 8 in 2024-03-12T03:28:32.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:31 smithi094 ceph-mon[29291]: pgmap v1016: 236 pgs: 236 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:28:32.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:31 smithi094 ceph-mon[29291]: osdmap e659: 8 total, 8 up, 8 in 2024-03-12T03:28:32.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:32.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:31 smithi094 ceph-mon[40484]: osdmap e658: 8 total, 8 up, 8 in 2024-03-12T03:28:32.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:31 smithi094 ceph-mon[40484]: pgmap v1016: 236 pgs: 236 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:28:32.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:31 smithi094 ceph-mon[40484]: osdmap e659: 8 total, 8 up, 8 in 2024-03-12T03:28:32.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:32.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:31 smithi151 ceph-mon[32923]: osdmap e658: 8 total, 8 up, 8 in 2024-03-12T03:28:32.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:31 smithi151 ceph-mon[32923]: pgmap v1016: 236 pgs: 236 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:28:32.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:31 smithi151 ceph-mon[32923]: osdmap e659: 8 total, 8 up, 8 in 2024-03-12T03:28:32.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:33 smithi094 ceph-mon[40484]: osdmap e660: 8 total, 8 up, 8 in 2024-03-12T03:28:33.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:33.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:33.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:33 smithi094 ceph-mon[29291]: osdmap e660: 8 total, 8 up, 8 in 2024-03-12T03:28:33.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:33.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:33.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:33 smithi151 ceph-mon[32923]: osdmap e660: 8 total, 8 up, 8 in 2024-03-12T03:28:33.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:34.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:34 smithi094 ceph-mon[29291]: pgmap v1019: 268 pgs: 11 unknown, 257 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:28:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:34 smithi094 ceph-mon[29291]: osdmap e661: 8 total, 8 up, 8 in 2024-03-12T03:28:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-132"}]: dispatch 2024-03-12T03:28:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:34 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:28:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:28:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:34 smithi094 ceph-mon[40484]: pgmap v1019: 268 pgs: 11 unknown, 257 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:28:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:34 smithi094 ceph-mon[40484]: osdmap e661: 8 total, 8 up, 8 in 2024-03-12T03:28:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-132"}]: dispatch 2024-03-12T03:28:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:34 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:28:34.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:28:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:34 smithi151 ceph-mon[32923]: pgmap v1019: 268 pgs: 11 unknown, 257 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:28:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:34 smithi151 ceph-mon[32923]: osdmap e661: 8 total, 8 up, 8 in 2024-03-12T03:28:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-132"}]: dispatch 2024-03-12T03:28:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:34 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:28:34.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:28:35.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-132"}]': finished 2024-03-12T03:28:35.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:35 smithi094 ceph-mon[29291]: osdmap e662: 8 total, 8 up, 8 in 2024-03-12T03:28:35.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-132", "mode": "writeback"}]: dispatch 2024-03-12T03:28:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-132"}]': finished 2024-03-12T03:28:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:35 smithi094 ceph-mon[40484]: osdmap e662: 8 total, 8 up, 8 in 2024-03-12T03:28:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-132", "mode": "writeback"}]: dispatch 2024-03-12T03:28:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-132"}]': finished 2024-03-12T03:28:35.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:35 smithi151 ceph-mon[32923]: osdmap e662: 8 total, 8 up, 8 in 2024-03-12T03:28:35.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-132", "mode": "writeback"}]: dispatch 2024-03-12T03:28:36.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:36 smithi094 ceph-mon[29291]: pgmap v1022: 268 pgs: 268 active+clean; 582 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:36.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:36 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:36.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-132", "mode": "writeback"}]': finished 2024-03-12T03:28:36.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:36 smithi094 ceph-mon[29291]: osdmap e663: 8 total, 8 up, 8 in 2024-03-12T03:28:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:36 smithi094 ceph-mon[40484]: pgmap v1022: 268 pgs: 268 active+clean; 582 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:36 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-132", "mode": "writeback"}]': finished 2024-03-12T03:28:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:36 smithi094 ceph-mon[40484]: osdmap e663: 8 total, 8 up, 8 in 2024-03-12T03:28:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:36 smithi151 ceph-mon[32923]: pgmap v1022: 268 pgs: 268 active+clean; 582 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:36 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-132", "mode": "writeback"}]': finished 2024-03-12T03:28:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:36 smithi151 ceph-mon[32923]: osdmap e663: 8 total, 8 up, 8 in 2024-03-12T03:28:37.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:37 smithi094 ceph-mon[29291]: osdmap e664: 8 total, 8 up, 8 in 2024-03-12T03:28:37.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:37 smithi094 ceph-mon[40484]: osdmap e664: 8 total, 8 up, 8 in 2024-03-12T03:28:37.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:37 smithi151 ceph-mon[32923]: osdmap e664: 8 total, 8 up, 8 in 2024-03-12T03:28:38.661 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:28:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:28:38.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:38 smithi094 ceph-mon[40484]: pgmap v1025: 268 pgs: 268 active+clean; 582 KiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:38 smithi094 ceph-mon[40484]: osdmap e665: 8 total, 8 up, 8 in 2024-03-12T03:28:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:38 smithi094 ceph-mon[29291]: pgmap v1025: 268 pgs: 268 active+clean; 582 KiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:38 smithi094 ceph-mon[29291]: osdmap e665: 8 total, 8 up, 8 in 2024-03-12T03:28:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:38 smithi151 ceph-mon[32923]: pgmap v1025: 268 pgs: 268 active+clean; 582 KiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:38 smithi151 ceph-mon[32923]: osdmap e665: 8 total, 8 up, 8 in 2024-03-12T03:28:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:39.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:39.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:39.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:39 smithi094 ceph-mon[29291]: osdmap e666: 8 total, 8 up, 8 in 2024-03-12T03:28:39.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-132"}]: dispatch 2024-03-12T03:28:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:39 smithi094 ceph-mon[40484]: osdmap e666: 8 total, 8 up, 8 in 2024-03-12T03:28:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-132"}]: dispatch 2024-03-12T03:28:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:39 smithi151 ceph-mon[32923]: osdmap e666: 8 total, 8 up, 8 in 2024-03-12T03:28:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-132"}]: dispatch 2024-03-12T03:28:40.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:40 smithi094 ceph-mon[40484]: pgmap v1028: 268 pgs: 268 active+clean; 582 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:28:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-132"}]': finished 2024-03-12T03:28:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:40 smithi094 ceph-mon[40484]: osdmap e667: 8 total, 8 up, 8 in 2024-03-12T03:28:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:40 smithi094 ceph-mon[29291]: pgmap v1028: 268 pgs: 268 active+clean; 582 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:28:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-132"}]': finished 2024-03-12T03:28:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:40 smithi094 ceph-mon[29291]: osdmap e667: 8 total, 8 up, 8 in 2024-03-12T03:28:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:28:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:28:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:28:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:40 smithi151 ceph-mon[32923]: pgmap v1028: 268 pgs: 268 active+clean; 582 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:28:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-132"}]': finished 2024-03-12T03:28:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:40 smithi151 ceph-mon[32923]: osdmap e667: 8 total, 8 up, 8 in 2024-03-12T03:28:41.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:41 smithi094 ceph-mon[40484]: osdmap e668: 8 total, 8 up, 8 in 2024-03-12T03:28:41.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:41.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:41 smithi094 ceph-mon[40484]: pgmap v1031: 268 pgs: 268 active+clean; 582 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:28:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:41 smithi094 ceph-mon[29291]: osdmap e668: 8 total, 8 up, 8 in 2024-03-12T03:28:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:41 smithi094 ceph-mon[29291]: pgmap v1031: 268 pgs: 268 active+clean; 582 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:28:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:41 smithi151 ceph-mon[32923]: osdmap e668: 8 total, 8 up, 8 in 2024-03-12T03:28:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:41 smithi151 ceph-mon[32923]: pgmap v1031: 268 pgs: 268 active+clean; 582 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:28:42.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:42.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:42 smithi094 ceph-mon[29291]: osdmap e669: 8 total, 8 up, 8 in 2024-03-12T03:28:42.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-132"}]: dispatch 2024-03-12T03:28:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:42 smithi094 ceph-mon[40484]: osdmap e669: 8 total, 8 up, 8 in 2024-03-12T03:28:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-132"}]: dispatch 2024-03-12T03:28:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:42 smithi151 ceph-mon[32923]: osdmap e669: 8 total, 8 up, 8 in 2024-03-12T03:28:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-132"}]: dispatch 2024-03-12T03:28:43.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:43 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:43.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-132"}]': finished 2024-03-12T03:28:43.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:43 smithi094 ceph-mon[29291]: osdmap e670: 8 total, 8 up, 8 in 2024-03-12T03:28:43.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:43 smithi094 ceph-mon[29291]: pgmap v1034: 268 pgs: 268 active+clean; 582 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:28:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:43 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-132"}]': finished 2024-03-12T03:28:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:43 smithi094 ceph-mon[40484]: osdmap e670: 8 total, 8 up, 8 in 2024-03-12T03:28:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:43 smithi094 ceph-mon[40484]: pgmap v1034: 268 pgs: 268 active+clean; 582 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:28:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:43 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-132"}]': finished 2024-03-12T03:28:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:43 smithi151 ceph-mon[32923]: osdmap e670: 8 total, 8 up, 8 in 2024-03-12T03:28:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:43 smithi151 ceph-mon[32923]: pgmap v1034: 268 pgs: 268 active+clean; 582 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-12T03:28:44.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:44 smithi094 ceph-mon[29291]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:28:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:44 smithi094 ceph-mon[29291]: osdmap e671: 8 total, 8 up, 8 in 2024-03-12T03:28:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:44 smithi094 ceph-mon[40484]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:28:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:44 smithi094 ceph-mon[40484]: osdmap e671: 8 total, 8 up, 8 in 2024-03-12T03:28:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:44 smithi151 ceph-mon[32923]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:28:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:44 smithi151 ceph-mon[32923]: osdmap e671: 8 total, 8 up, 8 in 2024-03-12T03:28:45.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:45 smithi094 ceph-mon[29291]: osdmap e672: 8 total, 8 up, 8 in 2024-03-12T03:28:45.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:45.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:45 smithi094 ceph-mon[29291]: pgmap v1037: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:45 smithi094 ceph-mon[40484]: osdmap e672: 8 total, 8 up, 8 in 2024-03-12T03:28:45.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:45.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:45 smithi094 ceph-mon[40484]: pgmap v1037: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:45 smithi151 ceph-mon[32923]: osdmap e672: 8 total, 8 up, 8 in 2024-03-12T03:28:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:45 smithi151 ceph-mon[32923]: pgmap v1037: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:46 smithi151 ceph-mon[32923]: osdmap e673: 8 total, 8 up, 8 in 2024-03-12T03:28:46.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:46 smithi094 ceph-mon[40484]: osdmap e673: 8 total, 8 up, 8 in 2024-03-12T03:28:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:46 smithi094 ceph-mon[29291]: osdmap e673: 8 total, 8 up, 8 in 2024-03-12T03:28:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:48.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:48.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[29291]: osdmap e674: 8 total, 8 up, 8 in 2024-03-12T03:28:48.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-134"}]: dispatch 2024-03-12T03:28:48.112 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[29291]: pgmap v1040: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:48.112 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-134"}]': finished 2024-03-12T03:28:48.112 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[29291]: osdmap e675: 8 total, 8 up, 8 in 2024-03-12T03:28:48.112 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-134", "mode": "writeback"}]: dispatch 2024-03-12T03:28:48.112 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:48.112 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[40484]: osdmap e674: 8 total, 8 up, 8 in 2024-03-12T03:28:48.112 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-134"}]: dispatch 2024-03-12T03:28:48.112 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[40484]: pgmap v1040: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:48.112 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-134"}]': finished 2024-03-12T03:28:48.113 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[40484]: osdmap e675: 8 total, 8 up, 8 in 2024-03-12T03:28:48.113 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-134", "mode": "writeback"}]: dispatch 2024-03-12T03:28:48.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:48.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:47 smithi151 ceph-mon[32923]: osdmap e674: 8 total, 8 up, 8 in 2024-03-12T03:28:48.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-134"}]: dispatch 2024-03-12T03:28:48.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:47 smithi151 ceph-mon[32923]: pgmap v1040: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:28:48.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-134"}]': finished 2024-03-12T03:28:48.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:47 smithi151 ceph-mon[32923]: osdmap e675: 8 total, 8 up, 8 in 2024-03-12T03:28:48.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-134", "mode": "writeback"}]: dispatch 2024-03-12T03:28:48.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:28:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:28:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:48 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-134", "mode": "writeback"}]': finished 2024-03-12T03:28:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:48 smithi094 ceph-mon[29291]: osdmap e676: 8 total, 8 up, 8 in 2024-03-12T03:28:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:48 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:48 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:48 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:49.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-134", "mode": "writeback"}]': finished 2024-03-12T03:28:49.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:48 smithi094 ceph-mon[40484]: osdmap e676: 8 total, 8 up, 8 in 2024-03-12T03:28:49.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:48 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:49.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:48 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:48 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-134", "mode": "writeback"}]': finished 2024-03-12T03:28:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:48 smithi151 ceph-mon[32923]: osdmap e676: 8 total, 8 up, 8 in 2024-03-12T03:28:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:48 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:49.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:48 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:50.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:50 smithi094 ceph-mon[29291]: pgmap v1043: 268 pgs: 268 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:50 smithi094 ceph-mon[29291]: osdmap e677: 8 total, 8 up, 8 in 2024-03-12T03:28:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:50 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-134"}]: dispatch 2024-03-12T03:28:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:50 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:28:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:28:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:50 smithi094 ceph-mon[40484]: pgmap v1043: 268 pgs: 268 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:50.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:50 smithi094 ceph-mon[40484]: osdmap e677: 8 total, 8 up, 8 in 2024-03-12T03:28:50.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:50 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-134"}]: dispatch 2024-03-12T03:28:50.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:50 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:28:50.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:28:50.711 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:28:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:28:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:28:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:50 smithi151 ceph-mon[32923]: pgmap v1043: 268 pgs: 268 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:50 smithi151 ceph-mon[32923]: osdmap e677: 8 total, 8 up, 8 in 2024-03-12T03:28:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:50 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-134"}]: dispatch 2024-03-12T03:28:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:50 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:28:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:28:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:51 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:51 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-134"}]': finished 2024-03-12T03:28:51.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:51 smithi094 ceph-mon[29291]: osdmap e678: 8 total, 8 up, 8 in 2024-03-12T03:28:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:51 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:51 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-134"}]': finished 2024-03-12T03:28:51.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:51 smithi094 ceph-mon[40484]: osdmap e678: 8 total, 8 up, 8 in 2024-03-12T03:28:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:51 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:51 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-134"}]': finished 2024-03-12T03:28:51.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:51 smithi151 ceph-mon[32923]: osdmap e678: 8 total, 8 up, 8 in 2024-03-12T03:28:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:52 smithi094 ceph-mon[29291]: pgmap v1046: 268 pgs: 268 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:52 smithi094 ceph-mon[29291]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:28:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:52 smithi094 ceph-mon[29291]: osdmap e679: 8 total, 8 up, 8 in 2024-03-12T03:28:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:52 smithi094 ceph-mon[40484]: pgmap v1046: 268 pgs: 268 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:52 smithi094 ceph-mon[40484]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:28:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:52 smithi094 ceph-mon[40484]: osdmap e679: 8 total, 8 up, 8 in 2024-03-12T03:28:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:52 smithi151 ceph-mon[32923]: pgmap v1046: 268 pgs: 268 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:52 smithi151 ceph-mon[32923]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:28:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:52 smithi151 ceph-mon[32923]: osdmap e679: 8 total, 8 up, 8 in 2024-03-12T03:28:53.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:53 smithi094 ceph-mon[29291]: osdmap e680: 8 total, 8 up, 8 in 2024-03-12T03:28:53.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:53 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:53 smithi094 ceph-mon[40484]: osdmap e680: 8 total, 8 up, 8 in 2024-03-12T03:28:53.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:53 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:53 smithi151 ceph-mon[32923]: osdmap e680: 8 total, 8 up, 8 in 2024-03-12T03:28:53.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:53 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:28:54.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:54 smithi094 ceph-mon[40484]: pgmap v1049: 268 pgs: 18 creating+peering, 14 unknown, 236 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:54.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:54.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:54 smithi094 ceph-mon[40484]: osdmap e681: 8 total, 8 up, 8 in 2024-03-12T03:28:54.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:54 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:54 smithi094 ceph-mon[29291]: pgmap v1049: 268 pgs: 18 creating+peering, 14 unknown, 236 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:54 smithi094 ceph-mon[29291]: osdmap e681: 8 total, 8 up, 8 in 2024-03-12T03:28:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:54 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:54 smithi151 ceph-mon[32923]: pgmap v1049: 268 pgs: 18 creating+peering, 14 unknown, 236 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:28:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:28:54.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:54 smithi151 ceph-mon[32923]: osdmap e681: 8 total, 8 up, 8 in 2024-03-12T03:28:54.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:54 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:28:55.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:55 smithi094 ceph-mon[29291]: osdmap e682: 8 total, 8 up, 8 in 2024-03-12T03:28:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-136"}]: dispatch 2024-03-12T03:28:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-136"}]': finished 2024-03-12T03:28:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:55 smithi094 ceph-mon[29291]: osdmap e683: 8 total, 8 up, 8 in 2024-03-12T03:28:55.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:55 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-136", "mode": "writeback"}]: dispatch 2024-03-12T03:28:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:55 smithi094 ceph-mon[40484]: osdmap e682: 8 total, 8 up, 8 in 2024-03-12T03:28:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-136"}]: dispatch 2024-03-12T03:28:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-136"}]': finished 2024-03-12T03:28:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:55 smithi094 ceph-mon[40484]: osdmap e683: 8 total, 8 up, 8 in 2024-03-12T03:28:55.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:55 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-136", "mode": "writeback"}]: dispatch 2024-03-12T03:28:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:28:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:55 smithi151 ceph-mon[32923]: osdmap e682: 8 total, 8 up, 8 in 2024-03-12T03:28:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-136"}]: dispatch 2024-03-12T03:28:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-136"}]': finished 2024-03-12T03:28:55.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:55 smithi151 ceph-mon[32923]: osdmap e683: 8 total, 8 up, 8 in 2024-03-12T03:28:55.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:55 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-136", "mode": "writeback"}]: dispatch 2024-03-12T03:28:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:56 smithi094 ceph-mon[40484]: pgmap v1052: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:28:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:56 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:56 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-136", "mode": "writeback"}]': finished 2024-03-12T03:28:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:56 smithi094 ceph-mon[40484]: osdmap e684: 8 total, 8 up, 8 in 2024-03-12T03:28:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:56 smithi094 ceph-mon[29291]: pgmap v1052: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:28:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:56 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:56.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:56 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-136", "mode": "writeback"}]': finished 2024-03-12T03:28:56.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:56 smithi094 ceph-mon[29291]: osdmap e684: 8 total, 8 up, 8 in 2024-03-12T03:28:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:56 smithi151 ceph-mon[32923]: pgmap v1052: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-12T03:28:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:56 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:28:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:56 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-136", "mode": "writeback"}]': finished 2024-03-12T03:28:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:56 smithi151 ceph-mon[32923]: osdmap e684: 8 total, 8 up, 8 in 2024-03-12T03:28:57.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:57 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:57.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:57 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:57.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:57 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:28:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:58 smithi094 ceph-mon[40484]: pgmap v1055: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-12T03:28:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:58 smithi094 ceph-mon[40484]: osdmap e685: 8 total, 8 up, 8 in 2024-03-12T03:28:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-136"}]: dispatch 2024-03-12T03:28:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:58 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:28:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:58 smithi094 ceph-mon[29291]: pgmap v1055: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-12T03:28:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:58 smithi094 ceph-mon[29291]: osdmap e685: 8 total, 8 up, 8 in 2024-03-12T03:28:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-136"}]: dispatch 2024-03-12T03:28:58.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:58 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:28:58.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:28:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:28:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:58 smithi151 ceph-mon[32923]: pgmap v1055: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-12T03:28:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:28:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:58 smithi151 ceph-mon[32923]: osdmap e685: 8 total, 8 up, 8 in 2024-03-12T03:28:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-136"}]: dispatch 2024-03-12T03:28:58.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:58 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:28:59.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:59 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:59.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-136"}]': finished 2024-03-12T03:28:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:59 smithi094 ceph-mon[40484]: osdmap e686: 8 total, 8 up, 8 in 2024-03-12T03:28:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:28:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:59 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-136"}]': finished 2024-03-12T03:28:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:59 smithi094 ceph-mon[29291]: osdmap e686: 8 total, 8 up, 8 in 2024-03-12T03:28:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:28:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:28:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:59 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:28:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-136"}]': finished 2024-03-12T03:28:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:59 smithi151 ceph-mon[32923]: osdmap e686: 8 total, 8 up, 8 in 2024-03-12T03:28:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:28:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:00.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:29:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:29:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:29:00.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:00 smithi094 ceph-mon[29291]: pgmap v1058: 268 pgs: 268 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-12T03:29:00.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:00 smithi094 ceph-mon[29291]: osdmap e687: 8 total, 8 up, 8 in 2024-03-12T03:29:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:00 smithi094 ceph-mon[40484]: pgmap v1058: 268 pgs: 268 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-12T03:29:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:00 smithi094 ceph-mon[40484]: osdmap e687: 8 total, 8 up, 8 in 2024-03-12T03:29:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:00 smithi151 ceph-mon[32923]: pgmap v1058: 268 pgs: 268 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-12T03:29:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:00 smithi151 ceph-mon[32923]: osdmap e687: 8 total, 8 up, 8 in 2024-03-12T03:29:01.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:01 smithi094 ceph-mon[29291]: osdmap e688: 8 total, 8 up, 8 in 2024-03-12T03:29:01.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:29:01.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:01 smithi094 ceph-mon[40484]: osdmap e688: 8 total, 8 up, 8 in 2024-03-12T03:29:01.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:29:01.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:01 smithi151 ceph-mon[32923]: osdmap e688: 8 total, 8 up, 8 in 2024-03-12T03:29:01.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:29:02.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:02 smithi094 ceph-mon[40484]: pgmap v1061: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-12T03:29:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:29:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:02 smithi094 ceph-mon[40484]: osdmap e689: 8 total, 8 up, 8 in 2024-03-12T03:29:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:29:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:02 smithi094 ceph-mon[29291]: pgmap v1061: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-12T03:29:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:29:02.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:02 smithi094 ceph-mon[29291]: osdmap e689: 8 total, 8 up, 8 in 2024-03-12T03:29:02.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:29:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:02 smithi151 ceph-mon[32923]: pgmap v1061: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-12T03:29:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:29:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:02 smithi151 ceph-mon[32923]: osdmap e689: 8 total, 8 up, 8 in 2024-03-12T03:29:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:29:03.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:29:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:03 smithi094 ceph-mon[29291]: osdmap e690: 8 total, 8 up, 8 in 2024-03-12T03:29:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:29:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:03 smithi094 ceph-mon[29291]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:29:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:29:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:29:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:03 smithi094 ceph-mon[40484]: osdmap e690: 8 total, 8 up, 8 in 2024-03-12T03:29:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:29:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:03 smithi094 ceph-mon[40484]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:29:03.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:29:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:29:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:03 smithi151 ceph-mon[32923]: osdmap e690: 8 total, 8 up, 8 in 2024-03-12T03:29:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:29:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:03 smithi151 ceph-mon[32923]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:29:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:29:04.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:04 smithi094 ceph-mon[29291]: pgmap v1064: 268 pgs: 20 unknown, 248 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:04 smithi094 ceph-mon[29291]: osdmap e691: 8 total, 8 up, 8 in 2024-03-12T03:29:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:29:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:29:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:04 smithi094 ceph-mon[40484]: pgmap v1064: 268 pgs: 20 unknown, 248 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:04 smithi094 ceph-mon[40484]: osdmap e691: 8 total, 8 up, 8 in 2024-03-12T03:29:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:29:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:29:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:04 smithi151 ceph-mon[32923]: pgmap v1064: 268 pgs: 20 unknown, 248 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:04 smithi151 ceph-mon[32923]: osdmap e691: 8 total, 8 up, 8 in 2024-03-12T03:29:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:29:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:29:05.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:05 smithi094 ceph-mon[40484]: osdmap e692: 8 total, 8 up, 8 in 2024-03-12T03:29:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:29:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:05 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:29:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-12T03:29:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:05 smithi094 ceph-mon[40484]: osdmap e693: 8 total, 8 up, 8 in 2024-03-12T03:29:05.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:05 smithi094 ceph-mon[29291]: osdmap e692: 8 total, 8 up, 8 in 2024-03-12T03:29:05.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:29:05.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:05 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:05.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:29:05.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-12T03:29:05.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:05 smithi094 ceph-mon[29291]: osdmap e693: 8 total, 8 up, 8 in 2024-03-12T03:29:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:05 smithi151 ceph-mon[32923]: osdmap e692: 8 total, 8 up, 8 in 2024-03-12T03:29:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-12T03:29:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:05 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:29:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-12T03:29:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:05 smithi151 ceph-mon[32923]: osdmap e693: 8 total, 8 up, 8 in 2024-03-12T03:29:06.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:06 smithi094 ceph-mon[40484]: pgmap v1067: 268 pgs: 268 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-138"}]: dispatch 2024-03-12T03:29:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:06 smithi094 ceph-mon[29291]: pgmap v1067: 268 pgs: 268 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-138"}]: dispatch 2024-03-12T03:29:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:06 smithi151 ceph-mon[32923]: pgmap v1067: 268 pgs: 268 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-138"}]: dispatch 2024-03-12T03:29:07.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-138"}]': finished 2024-03-12T03:29:07.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:07 smithi094 ceph-mon[40484]: osdmap e694: 8 total, 8 up, 8 in 2024-03-12T03:29:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-138"}]': finished 2024-03-12T03:29:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:07 smithi094 ceph-mon[29291]: osdmap e694: 8 total, 8 up, 8 in 2024-03-12T03:29:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-138"}]': finished 2024-03-12T03:29:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:07 smithi151 ceph-mon[32923]: osdmap e694: 8 total, 8 up, 8 in 2024-03-12T03:29:08.646 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:29:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:29:08.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:08 smithi094 ceph-mon[40484]: pgmap v1070: 268 pgs: 268 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:29:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:08 smithi094 ceph-mon[40484]: osdmap e695: 8 total, 8 up, 8 in 2024-03-12T03:29:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:08 smithi094 ceph-mon[29291]: pgmap v1070: 268 pgs: 268 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:29:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:08 smithi094 ceph-mon[29291]: osdmap e695: 8 total, 8 up, 8 in 2024-03-12T03:29:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:08 smithi151 ceph-mon[32923]: pgmap v1070: 268 pgs: 268 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:29:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:08 smithi151 ceph-mon[32923]: osdmap e695: 8 total, 8 up, 8 in 2024-03-12T03:29:09.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:09 smithi094 ceph-mon[40484]: osdmap e696: 8 total, 8 up, 8 in 2024-03-12T03:29:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:29:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:09 smithi094 ceph-mon[29291]: osdmap e696: 8 total, 8 up, 8 in 2024-03-12T03:29:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:29:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:09 smithi151 ceph-mon[32923]: osdmap e696: 8 total, 8 up, 8 in 2024-03-12T03:29:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:29:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:29:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:29:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:29:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:10 smithi094 ceph-mon[40484]: pgmap v1073: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:29:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:10 smithi094 ceph-mon[40484]: osdmap e697: 8 total, 8 up, 8 in 2024-03-12T03:29:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:29:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:10 smithi094 ceph-mon[29291]: pgmap v1073: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:10.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:29:10.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:10 smithi094 ceph-mon[29291]: osdmap e697: 8 total, 8 up, 8 in 2024-03-12T03:29:10.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:29:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:10 smithi151 ceph-mon[32923]: pgmap v1073: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:29:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:10 smithi151 ceph-mon[32923]: osdmap e697: 8 total, 8 up, 8 in 2024-03-12T03:29:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:29:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:29:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:11 smithi094 ceph-mon[29291]: osdmap e698: 8 total, 8 up, 8 in 2024-03-12T03:29:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-12T03:29:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_count","val": "3"}]': finished 2024-03-12T03:29:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:11 smithi094 ceph-mon[29291]: osdmap e699: 8 total, 8 up, 8 in 2024-03-12T03:29:11.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:11 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-12T03:29:11.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:29:11.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:11 smithi094 ceph-mon[40484]: osdmap e698: 8 total, 8 up, 8 in 2024-03-12T03:29:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-12T03:29:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_count","val": "3"}]': finished 2024-03-12T03:29:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:11 smithi094 ceph-mon[40484]: osdmap e699: 8 total, 8 up, 8 in 2024-03-12T03:29:11.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:11 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-12T03:29:11.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:29:11.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:11 smithi151 ceph-mon[32923]: osdmap e698: 8 total, 8 up, 8 in 2024-03-12T03:29:11.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-12T03:29:11.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_count","val": "3"}]': finished 2024-03-12T03:29:11.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:11 smithi151 ceph-mon[32923]: osdmap e699: 8 total, 8 up, 8 in 2024-03-12T03:29:11.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:11 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-12T03:29:12.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:12 smithi094 ceph-mon[40484]: pgmap v1076: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:12 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:29:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_period","val": "3"}]': finished 2024-03-12T03:29:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:12 smithi094 ceph-mon[40484]: osdmap e700: 8 total, 8 up, 8 in 2024-03-12T03:29:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:12 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:29:12.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:12 smithi094 ceph-mon[29291]: pgmap v1076: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:12.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:12 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:29:12.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_period","val": "3"}]': finished 2024-03-12T03:29:12.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:12 smithi094 ceph-mon[29291]: osdmap e700: 8 total, 8 up, 8 in 2024-03-12T03:29:12.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:12 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:29:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:12 smithi151 ceph-mon[32923]: pgmap v1076: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:12 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:29:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_period","val": "3"}]': finished 2024-03-12T03:29:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:12 smithi151 ceph-mon[32923]: osdmap e700: 8 total, 8 up, 8 in 2024-03-12T03:29:12.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:12 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:29:14.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:14 smithi094 ceph-mon[40484]: pgmap v1079: 268 pgs: 21 unknown, 247 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:29:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:14 smithi094 ceph-mon[40484]: osdmap e701: 8 total, 8 up, 8 in 2024-03-12T03:29:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:14 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-12T03:29:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:14 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:14 smithi094 ceph-mon[29291]: pgmap v1079: 268 pgs: 21 unknown, 247 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:29:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:14 smithi094 ceph-mon[29291]: osdmap e701: 8 total, 8 up, 8 in 2024-03-12T03:29:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:14 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-12T03:29:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:14 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:14 smithi151 ceph-mon[32923]: pgmap v1079: 268 pgs: 21 unknown, 247 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:29:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:14 smithi151 ceph-mon[32923]: osdmap e701: 8 total, 8 up, 8 in 2024-03-12T03:29:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:14 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-12T03:29:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:14 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:15.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-12T03:29:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[29291]: osdmap e702: 8 total, 8 up, 8 in 2024-03-12T03:29:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[29291]: pgmap v1082: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:15.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-12T03:29:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[40484]: osdmap e702: 8 total, 8 up, 8 in 2024-03-12T03:29:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[40484]: pgmap v1082: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:15.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:15.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:15.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:15.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:15 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:15 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-12T03:29:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:15 smithi151 ceph-mon[32923]: osdmap e702: 8 total, 8 up, 8 in 2024-03-12T03:29:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:15 smithi151 ceph-mon[32923]: pgmap v1082: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:15 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:15 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:15 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:15.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:15 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:15.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:15 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:18.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:17 smithi094 ceph-mon[29291]: pgmap v1083: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 996 B/s rd, 0 op/s 2024-03-12T03:29:18.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:17 smithi094 ceph-mon[40484]: pgmap v1083: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 996 B/s rd, 0 op/s 2024-03-12T03:29:18.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:17 smithi151 ceph-mon[32923]: pgmap v1083: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 996 B/s rd, 0 op/s 2024-03-12T03:29:18.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:29:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:29:19.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:20 smithi094 ceph-mon[29291]: pgmap v1084: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 5.4 KiB/s wr, 2 op/s 2024-03-12T03:29:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:29:20.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:29:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:29:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:29:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:20 smithi094 ceph-mon[40484]: pgmap v1084: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 5.4 KiB/s wr, 2 op/s 2024-03-12T03:29:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:29:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:20 smithi151 ceph-mon[32923]: pgmap v1084: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 5.4 KiB/s wr, 2 op/s 2024-03-12T03:29:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:29:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:22 smithi094 ceph-mon[29291]: pgmap v1085: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-03-12T03:29:22.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:22 smithi094 ceph-mon[40484]: pgmap v1085: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-03-12T03:29:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:22 smithi151 ceph-mon[32923]: pgmap v1085: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-03-12T03:29:24.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:24 smithi094 ceph-mon[29291]: pgmap v1086: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.6 KiB/s wr, 1 op/s 2024-03-12T03:29:24.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:24 smithi094 ceph-mon[40484]: pgmap v1086: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.6 KiB/s wr, 1 op/s 2024-03-12T03:29:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:24 smithi151 ceph-mon[32923]: pgmap v1086: 268 pgs: 268 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.6 KiB/s wr, 1 op/s 2024-03-12T03:29:26.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:26 smithi094 ceph-mon[29291]: pgmap v1087: 268 pgs: 268 active+clean; 4.5 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 7.3 KiB/s wr, 2 op/s 2024-03-12T03:29:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:26 smithi094 ceph-mon[40484]: pgmap v1087: 268 pgs: 268 active+clean; 4.5 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 7.3 KiB/s wr, 2 op/s 2024-03-12T03:29:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:26 smithi151 ceph-mon[32923]: pgmap v1087: 268 pgs: 268 active+clean; 4.5 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 7.3 KiB/s wr, 2 op/s 2024-03-12T03:29:27.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:27.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-140"}]: dispatch 2024-03-12T03:29:27.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:27.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-140"}]: dispatch 2024-03-12T03:29:27.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:27.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-140"}]: dispatch 2024-03-12T03:29:28.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:28 smithi094 ceph-mon[29291]: pgmap v1088: 268 pgs: 268 active+clean; 4.5 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 6.2 KiB/s wr, 1 op/s 2024-03-12T03:29:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-140"}]': finished 2024-03-12T03:29:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:28 smithi094 ceph-mon[29291]: osdmap e703: 8 total, 8 up, 8 in 2024-03-12T03:29:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:28 smithi094 ceph-mon[40484]: pgmap v1088: 268 pgs: 268 active+clean; 4.5 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 6.2 KiB/s wr, 1 op/s 2024-03-12T03:29:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-140"}]': finished 2024-03-12T03:29:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:28 smithi094 ceph-mon[40484]: osdmap e703: 8 total, 8 up, 8 in 2024-03-12T03:29:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:28 smithi151 ceph-mon[32923]: pgmap v1088: 268 pgs: 268 active+clean; 4.5 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 6.2 KiB/s wr, 1 op/s 2024-03-12T03:29:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-140"}]': finished 2024-03-12T03:29:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:28 smithi151 ceph-mon[32923]: osdmap e703: 8 total, 8 up, 8 in 2024-03-12T03:29:28.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:29:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:29:29.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:29 smithi094 ceph-mon[40484]: osdmap e704: 8 total, 8 up, 8 in 2024-03-12T03:29:29.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:29 smithi094 ceph-mon[29291]: osdmap e704: 8 total, 8 up, 8 in 2024-03-12T03:29:29.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:29 smithi151 ceph-mon[32923]: osdmap e704: 8 total, 8 up, 8 in 2024-03-12T03:29:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:30.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:29:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:29:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:29:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:30 smithi094 ceph-mon[40484]: pgmap v1091: 236 pgs: 236 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:30 smithi094 ceph-mon[40484]: osdmap e705: 8 total, 8 up, 8 in 2024-03-12T03:29:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:29:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:30 smithi094 ceph-mon[29291]: pgmap v1091: 236 pgs: 236 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:30 smithi094 ceph-mon[29291]: osdmap e705: 8 total, 8 up, 8 in 2024-03-12T03:29:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:29:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:30 smithi151 ceph-mon[32923]: pgmap v1091: 236 pgs: 236 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:30 smithi151 ceph-mon[32923]: osdmap e705: 8 total, 8 up, 8 in 2024-03-12T03:29:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:29:31.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:29:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:31 smithi094 ceph-mon[29291]: osdmap e706: 8 total, 8 up, 8 in 2024-03-12T03:29:31.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:31 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:29:31.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:29:31.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:31 smithi094 ceph-mon[40484]: osdmap e706: 8 total, 8 up, 8 in 2024-03-12T03:29:31.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:31 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:29:31.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:29:31.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:31 smithi151 ceph-mon[32923]: osdmap e706: 8 total, 8 up, 8 in 2024-03-12T03:29:31.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:31 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:29:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:32 smithi094 ceph-mon[29291]: pgmap v1094: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:29:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:32 smithi094 ceph-mon[29291]: osdmap e707: 8 total, 8 up, 8 in 2024-03-12T03:29:32.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-142"}]: dispatch 2024-03-12T03:29:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:32 smithi094 ceph-mon[40484]: pgmap v1094: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:29:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:32 smithi094 ceph-mon[40484]: osdmap e707: 8 total, 8 up, 8 in 2024-03-12T03:29:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-142"}]: dispatch 2024-03-12T03:29:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:32 smithi151 ceph-mon[32923]: pgmap v1094: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:29:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:32 smithi151 ceph-mon[32923]: osdmap e707: 8 total, 8 up, 8 in 2024-03-12T03:29:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-142"}]: dispatch 2024-03-12T03:29:33.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-142"}]': finished 2024-03-12T03:29:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[29291]: osdmap e708: 8 total, 8 up, 8 in 2024-03-12T03:29:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-142", "mode": "writeback"}]: dispatch 2024-03-12T03:29:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:29:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-142", "mode": "writeback"}]': finished 2024-03-12T03:29:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[29291]: osdmap e709: 8 total, 8 up, 8 in 2024-03-12T03:29:33.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:29:33.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-142"}]': finished 2024-03-12T03:29:33.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[40484]: osdmap e708: 8 total, 8 up, 8 in 2024-03-12T03:29:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-142", "mode": "writeback"}]: dispatch 2024-03-12T03:29:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:29:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-142", "mode": "writeback"}]': finished 2024-03-12T03:29:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[40484]: osdmap e709: 8 total, 8 up, 8 in 2024-03-12T03:29:33.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:29:33.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-142"}]': finished 2024-03-12T03:29:33.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:33 smithi151 ceph-mon[32923]: osdmap e708: 8 total, 8 up, 8 in 2024-03-12T03:29:33.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-142", "mode": "writeback"}]: dispatch 2024-03-12T03:29:33.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:33 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:29:33.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-142", "mode": "writeback"}]': finished 2024-03-12T03:29:33.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:33 smithi151 ceph-mon[32923]: osdmap e709: 8 total, 8 up, 8 in 2024-03-12T03:29:33.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:29:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:34 smithi094 ceph-mon[40484]: pgmap v1097: 268 pgs: 20 unknown, 248 active+clean; 4.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 3 op/s 2024-03-12T03:29:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:29:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:34 smithi094 ceph-mon[40484]: osdmap e710: 8 total, 8 up, 8 in 2024-03-12T03:29:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:34 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:29:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:34 smithi094 ceph-mon[29291]: pgmap v1097: 268 pgs: 20 unknown, 248 active+clean; 4.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 3 op/s 2024-03-12T03:29:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:29:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:34 smithi094 ceph-mon[29291]: osdmap e710: 8 total, 8 up, 8 in 2024-03-12T03:29:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:34 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:29:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:34 smithi151 ceph-mon[32923]: pgmap v1097: 268 pgs: 20 unknown, 248 active+clean; 4.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 3 op/s 2024-03-12T03:29:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:29:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:34 smithi151 ceph-mon[32923]: osdmap e710: 8 total, 8 up, 8 in 2024-03-12T03:29:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:34 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:29:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:35 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:29:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:29:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:35 smithi094 ceph-mon[40484]: osdmap e711: 8 total, 8 up, 8 in 2024-03-12T03:29:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:29:35.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:35 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:35.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:29:35.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:29:35.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:35 smithi094 ceph-mon[29291]: osdmap e711: 8 total, 8 up, 8 in 2024-03-12T03:29:35.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:29:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:35 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:29:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:29:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:29:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:35 smithi151 ceph-mon[32923]: osdmap e711: 8 total, 8 up, 8 in 2024-03-12T03:29:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:29:36.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:36 smithi094 ceph-mon[29291]: pgmap v1100: 268 pgs: 268 active+clean; 4.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-12T03:29:36.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:36 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:29:36.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:29:36.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:36 smithi094 ceph-mon[29291]: osdmap e712: 8 total, 8 up, 8 in 2024-03-12T03:29:36.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:29:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:36 smithi094 ceph-mon[40484]: pgmap v1100: 268 pgs: 268 active+clean; 4.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-12T03:29:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:36 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:29:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:29:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:36 smithi094 ceph-mon[40484]: osdmap e712: 8 total, 8 up, 8 in 2024-03-12T03:29:36.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:29:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:36 smithi151 ceph-mon[32923]: pgmap v1100: 268 pgs: 268 active+clean; 4.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-12T03:29:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:36 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:29:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:29:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:36 smithi151 ceph-mon[32923]: osdmap e712: 8 total, 8 up, 8 in 2024-03-12T03:29:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:29:38.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:38 smithi094 ceph-mon[29291]: pgmap v1103: 268 pgs: 268 active+clean; 4.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-12T03:29:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-12T03:29:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:38 smithi094 ceph-mon[29291]: osdmap e713: 8 total, 8 up, 8 in 2024-03-12T03:29:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:38 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-12T03:29:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:38 smithi094 ceph-mon[40484]: pgmap v1103: 268 pgs: 268 active+clean; 4.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-12T03:29:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-12T03:29:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:38 smithi094 ceph-mon[40484]: osdmap e713: 8 total, 8 up, 8 in 2024-03-12T03:29:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:38 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-12T03:29:38.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:29:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:29:38.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:38 smithi151 ceph-mon[32923]: pgmap v1103: 268 pgs: 268 active+clean; 4.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-12T03:29:38.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-12T03:29:38.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:38 smithi151 ceph-mon[32923]: osdmap e713: 8 total, 8 up, 8 in 2024-03-12T03:29:38.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:38 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-12T03:29:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-12T03:29:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:39 smithi094 ceph-mon[29291]: osdmap e714: 8 total, 8 up, 8 in 2024-03-12T03:29:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-12T03:29:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-12T03:29:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:39 smithi094 ceph-mon[29291]: osdmap e715: 8 total, 8 up, 8 in 2024-03-12T03:29:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-12T03:29:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:39 smithi094 ceph-mon[40484]: osdmap e714: 8 total, 8 up, 8 in 2024-03-12T03:29:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-12T03:29:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-12T03:29:39.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:39 smithi094 ceph-mon[40484]: osdmap e715: 8 total, 8 up, 8 in 2024-03-12T03:29:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-12T03:29:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:39 smithi151 ceph-mon[32923]: osdmap e714: 8 total, 8 up, 8 in 2024-03-12T03:29:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-12T03:29:39.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:39.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-12T03:29:39.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:39 smithi151 ceph-mon[32923]: osdmap e715: 8 total, 8 up, 8 in 2024-03-12T03:29:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:40 smithi094 ceph-mon[29291]: pgmap v1106: 268 pgs: 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:40.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:40 smithi094 ceph-mon[40484]: pgmap v1106: 268 pgs: 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:40.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:40.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:29:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:29:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:29:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:40 smithi151 ceph-mon[32923]: pgmap v1106: 268 pgs: 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:41.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:29:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:41 smithi094 ceph-mon[40484]: osdmap e716: 8 total, 8 up, 8 in 2024-03-12T03:29:41.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142"}]: dispatch 2024-03-12T03:29:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:29:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:41 smithi094 ceph-mon[29291]: osdmap e716: 8 total, 8 up, 8 in 2024-03-12T03:29:41.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142"}]: dispatch 2024-03-12T03:29:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:29:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:41 smithi151 ceph-mon[32923]: osdmap e716: 8 total, 8 up, 8 in 2024-03-12T03:29:41.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142"}]: dispatch 2024-03-12T03:29:42.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:42 smithi094 ceph-mon[40484]: pgmap v1109: 268 pgs: 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142"}]': finished 2024-03-12T03:29:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:42 smithi094 ceph-mon[40484]: osdmap e717: 8 total, 8 up, 8 in 2024-03-12T03:29:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:42 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142"}]: dispatch 2024-03-12T03:29:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:42 smithi094 ceph-mon[29291]: pgmap v1109: 268 pgs: 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142"}]': finished 2024-03-12T03:29:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:42 smithi094 ceph-mon[29291]: osdmap e717: 8 total, 8 up, 8 in 2024-03-12T03:29:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:42 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142"}]: dispatch 2024-03-12T03:29:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:42 smithi151 ceph-mon[32923]: pgmap v1109: 268 pgs: 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142"}]': finished 2024-03-12T03:29:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:42 smithi151 ceph-mon[32923]: osdmap e717: 8 total, 8 up, 8 in 2024-03-12T03:29:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:42.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:42 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-142"}]: dispatch 2024-03-12T03:29:43.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:43 smithi094 ceph-mon[29291]: osdmap e718: 8 total, 8 up, 8 in 2024-03-12T03:29:43.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:43 smithi094 ceph-mon[29291]: osdmap e719: 8 total, 8 up, 8 in 2024-03-12T03:29:43.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:43 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:29:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:43 smithi094 ceph-mon[40484]: osdmap e718: 8 total, 8 up, 8 in 2024-03-12T03:29:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:43 smithi094 ceph-mon[40484]: osdmap e719: 8 total, 8 up, 8 in 2024-03-12T03:29:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:43 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:29:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:43 smithi151 ceph-mon[32923]: osdmap e718: 8 total, 8 up, 8 in 2024-03-12T03:29:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:43 smithi151 ceph-mon[32923]: osdmap e719: 8 total, 8 up, 8 in 2024-03-12T03:29:43.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:43 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:29:44.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:44 smithi094 ceph-mon[29291]: pgmap v1112: 236 pgs: 236 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:29:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:44 smithi094 ceph-mon[29291]: osdmap e720: 8 total, 8 up, 8 in 2024-03-12T03:29:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:29:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:44 smithi094 ceph-mon[40484]: pgmap v1112: 236 pgs: 236 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:29:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:44 smithi094 ceph-mon[40484]: osdmap e720: 8 total, 8 up, 8 in 2024-03-12T03:29:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:29:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:44 smithi151 ceph-mon[32923]: pgmap v1112: 236 pgs: 236 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:29:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:29:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:44 smithi151 ceph-mon[32923]: osdmap e720: 8 total, 8 up, 8 in 2024-03-12T03:29:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:29:46.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:46 smithi094 ceph-mon[29291]: pgmap v1115: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:29:46.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:29:46.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:46 smithi094 ceph-mon[29291]: osdmap e721: 8 total, 8 up, 8 in 2024-03-12T03:29:46.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-144"}]: dispatch 2024-03-12T03:29:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:46 smithi094 ceph-mon[40484]: pgmap v1115: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:29:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:29:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:46 smithi094 ceph-mon[40484]: osdmap e721: 8 total, 8 up, 8 in 2024-03-12T03:29:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-144"}]: dispatch 2024-03-12T03:29:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:46 smithi151 ceph-mon[32923]: pgmap v1115: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:29:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:29:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:46 smithi151 ceph-mon[32923]: osdmap e721: 8 total, 8 up, 8 in 2024-03-12T03:29:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-144"}]: dispatch 2024-03-12T03:29:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-144"}]': finished 2024-03-12T03:29:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:47 smithi094 ceph-mon[29291]: osdmap e722: 8 total, 8 up, 8 in 2024-03-12T03:29:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-144", "mode": "readproxy"}]: dispatch 2024-03-12T03:29:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:47 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:29:47.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:47 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-144", "mode": "readproxy"}]': finished 2024-03-12T03:29:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-144"}]': finished 2024-03-12T03:29:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:47 smithi094 ceph-mon[40484]: osdmap e722: 8 total, 8 up, 8 in 2024-03-12T03:29:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-144", "mode": "readproxy"}]: dispatch 2024-03-12T03:29:47.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:47 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:29:47.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:47 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-144", "mode": "readproxy"}]': finished 2024-03-12T03:29:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-144"}]': finished 2024-03-12T03:29:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:47 smithi151 ceph-mon[32923]: osdmap e722: 8 total, 8 up, 8 in 2024-03-12T03:29:47.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-144", "mode": "readproxy"}]: dispatch 2024-03-12T03:29:47.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:47 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:29:47.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:47 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-144", "mode": "readproxy"}]': finished 2024-03-12T03:29:48.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:48 smithi094 ceph-mon[29291]: pgmap v1118: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:29:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:48 smithi094 ceph-mon[29291]: osdmap e723: 8 total, 8 up, 8 in 2024-03-12T03:29:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:48 smithi094 ceph-mon[40484]: pgmap v1118: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:29:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:48 smithi094 ceph-mon[40484]: osdmap e723: 8 total, 8 up, 8 in 2024-03-12T03:29:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:48 smithi151 ceph-mon[32923]: pgmap v1118: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:29:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:48 smithi151 ceph-mon[32923]: osdmap e723: 8 total, 8 up, 8 in 2024-03-12T03:29:48.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:29:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:29:49.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:49.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:50.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:50 smithi094 ceph-mon[29291]: pgmap v1120: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 231 B/s wr, 1 op/s 2024-03-12T03:29:50.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:29:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:50 smithi094 ceph-mon[40484]: pgmap v1120: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 231 B/s wr, 1 op/s 2024-03-12T03:29:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:29:50.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:29:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:29:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:29:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:50 smithi151 ceph-mon[32923]: pgmap v1120: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 231 B/s wr, 1 op/s 2024-03-12T03:29:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:29:52.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:52 smithi094 ceph-mon[29291]: pgmap v1121: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-12T03:29:52.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:52 smithi094 ceph-mon[40484]: pgmap v1121: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-12T03:29:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:52 smithi151 ceph-mon[32923]: pgmap v1121: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-12T03:29:54.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:54 smithi094 ceph-mon[29291]: pgmap v1122: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 137 B/s wr, 1 op/s 2024-03-12T03:29:54.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:54 smithi094 ceph-mon[40484]: pgmap v1122: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 137 B/s wr, 1 op/s 2024-03-12T03:29:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:54 smithi151 ceph-mon[32923]: pgmap v1122: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 137 B/s wr, 1 op/s 2024-03-12T03:29:56.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:56 smithi094 ceph-mon[29291]: pgmap v1123: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 121 B/s wr, 1 op/s 2024-03-12T03:29:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:56 smithi094 ceph-mon[40484]: pgmap v1123: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 121 B/s wr, 1 op/s 2024-03-12T03:29:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:56 smithi151 ceph-mon[32923]: pgmap v1123: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 121 B/s wr, 1 op/s 2024-03-12T03:29:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:58 smithi094 ceph-mon[29291]: pgmap v1124: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-12T03:29:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:58 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:58.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:58 smithi094 ceph-mon[40484]: pgmap v1124: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-12T03:29:58.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:58 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:58 smithi151 ceph-mon[32923]: pgmap v1124: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-12T03:29:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:58 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:29:58.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:29:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:29:59.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:29:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:59 smithi094 ceph-mon[29291]: osdmap e724: 8 total, 8 up, 8 in 2024-03-12T03:29:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144"}]: dispatch 2024-03-12T03:29:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:59 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:29:59.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:29:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144"}]': finished 2024-03-12T03:29:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:29:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:59 smithi094 ceph-mon[40484]: osdmap e724: 8 total, 8 up, 8 in 2024-03-12T03:29:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144"}]: dispatch 2024-03-12T03:29:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:59 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:29:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:29:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144"}]': finished 2024-03-12T03:29:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:29:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:59 smithi151 ceph-mon[32923]: osdmap e724: 8 total, 8 up, 8 in 2024-03-12T03:29:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144"}]: dispatch 2024-03-12T03:29:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:29:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:59 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:29:59.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:29:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144"}]': finished 2024-03-12T03:30:00.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:30:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:30:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[40484]: pgmap v1126: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:30:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[40484]: osdmap e725: 8 total, 8 up, 8 in 2024-03-12T03:30:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144"}]: dispatch 2024-03-12T03:30:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[40484]: Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-12T03:30:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[40484]: [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-12T03:30:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[40484]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-12T03:30:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[40484]: application not enabled on pool 'ListObjectssmithi094-79253-1' 2024-03-12T03:30:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[40484]: application not enabled on pool 'SetOpFlagssmithi094-79449-1' 2024-03-12T03:30:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[40484]: application not enabled on pool 'test-rados-api-smithi094-78452-111' 2024-03-12T03:30:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[40484]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-12T03:30:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[29291]: pgmap v1126: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:30:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[29291]: osdmap e725: 8 total, 8 up, 8 in 2024-03-12T03:30:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144"}]: dispatch 2024-03-12T03:30:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[29291]: Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-12T03:30:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[29291]: [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-12T03:30:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[29291]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-12T03:30:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[29291]: application not enabled on pool 'ListObjectssmithi094-79253-1' 2024-03-12T03:30:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[29291]: application not enabled on pool 'SetOpFlagssmithi094-79449-1' 2024-03-12T03:30:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[29291]: application not enabled on pool 'test-rados-api-smithi094-78452-111' 2024-03-12T03:30:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:00 smithi094 ceph-mon[29291]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-12T03:30:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:00 smithi151 ceph-mon[32923]: pgmap v1126: 268 pgs: 268 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:30:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:00 smithi151 ceph-mon[32923]: osdmap e725: 8 total, 8 up, 8 in 2024-03-12T03:30:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-144"}]: dispatch 2024-03-12T03:30:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:00 smithi151 ceph-mon[32923]: Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-12T03:30:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:00 smithi151 ceph-mon[32923]: [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-12T03:30:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:00 smithi151 ceph-mon[32923]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-12T03:30:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:00 smithi151 ceph-mon[32923]: application not enabled on pool 'ListObjectssmithi094-79253-1' 2024-03-12T03:30:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:00 smithi151 ceph-mon[32923]: application not enabled on pool 'SetOpFlagssmithi094-79449-1' 2024-03-12T03:30:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:00 smithi151 ceph-mon[32923]: application not enabled on pool 'test-rados-api-smithi094-78452-111' 2024-03-12T03:30:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:00 smithi151 ceph-mon[32923]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-12T03:30:01.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:01 smithi094 ceph-mon[40484]: osdmap e726: 8 total, 8 up, 8 in 2024-03-12T03:30:01.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:01 smithi094 ceph-mon[29291]: osdmap e726: 8 total, 8 up, 8 in 2024-03-12T03:30:01.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:01 smithi151 ceph-mon[32923]: osdmap e726: 8 total, 8 up, 8 in 2024-03-12T03:30:02.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:02 smithi094 ceph-mon[29291]: pgmap v1129: 236 pgs: 236 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:30:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:02 smithi094 ceph-mon[29291]: osdmap e727: 8 total, 8 up, 8 in 2024-03-12T03:30:02.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:02 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:02 smithi094 ceph-mon[40484]: pgmap v1129: 236 pgs: 236 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:30:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:02 smithi094 ceph-mon[40484]: osdmap e727: 8 total, 8 up, 8 in 2024-03-12T03:30:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:02 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:02 smithi151 ceph-mon[32923]: pgmap v1129: 236 pgs: 236 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:30:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:02 smithi151 ceph-mon[32923]: osdmap e727: 8 total, 8 up, 8 in 2024-03-12T03:30:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:02 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:03 smithi094 ceph-mon[29291]: osdmap e728: 8 total, 8 up, 8 in 2024-03-12T03:30:03.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:03 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:30:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:03 smithi094 ceph-mon[40484]: osdmap e728: 8 total, 8 up, 8 in 2024-03-12T03:30:03.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:03 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:30:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:03.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:03 smithi151 ceph-mon[32923]: osdmap e728: 8 total, 8 up, 8 in 2024-03-12T03:30:03.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:03 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:30:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:04 smithi151 ceph-mon[32923]: pgmap v1132: 268 pgs: 13 unknown, 255 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:30:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:30:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:04 smithi151 ceph-mon[32923]: osdmap e729: 8 total, 8 up, 8 in 2024-03-12T03:30:04.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:04 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-146"}]: dispatch 2024-03-12T03:30:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:04 smithi094 ceph-mon[29291]: pgmap v1132: 268 pgs: 13 unknown, 255 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:30:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:30:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:04 smithi094 ceph-mon[29291]: osdmap e729: 8 total, 8 up, 8 in 2024-03-12T03:30:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:04 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-146"}]: dispatch 2024-03-12T03:30:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:04 smithi094 ceph-mon[40484]: pgmap v1132: 268 pgs: 13 unknown, 255 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-12T03:30:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:30:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:04 smithi094 ceph-mon[40484]: osdmap e729: 8 total, 8 up, 8 in 2024-03-12T03:30:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:04 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-146"}]: dispatch 2024-03-12T03:30:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-146"}]': finished 2024-03-12T03:30:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:05 smithi151 ceph-mon[32923]: osdmap e730: 8 total, 8 up, 8 in 2024-03-12T03:30:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-146", "mode": "writeback"}]: dispatch 2024-03-12T03:30:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:05 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:30:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:05 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:30:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-146", "mode": "writeback"}]': finished 2024-03-12T03:30:05.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-146"}]': finished 2024-03-12T03:30:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[29291]: osdmap e730: 8 total, 8 up, 8 in 2024-03-12T03:30:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-146", "mode": "writeback"}]: dispatch 2024-03-12T03:30:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:30:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:30:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-146", "mode": "writeback"}]': finished 2024-03-12T03:30:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi094-78452-111", "overlaypool": "test-rados-api-smithi094-78452-146"}]': finished 2024-03-12T03:30:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[40484]: osdmap e730: 8 total, 8 up, 8 in 2024-03-12T03:30:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-146", "mode": "writeback"}]: dispatch 2024-03-12T03:30:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:30:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-12T03:30:05.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi094-78452-146", "mode": "writeback"}]': finished 2024-03-12T03:30:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:06 smithi094 ceph-mon[29291]: pgmap v1135: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:30:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:06 smithi094 ceph-mon[29291]: osdmap e731: 8 total, 8 up, 8 in 2024-03-12T03:30:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:30:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:06 smithi094 ceph-mon[40484]: pgmap v1135: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:30:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:06 smithi094 ceph-mon[40484]: osdmap e731: 8 total, 8 up, 8 in 2024-03-12T03:30:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:30:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:06 smithi151 ceph-mon[32923]: pgmap v1135: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:30:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:06 smithi151 ceph-mon[32923]: osdmap e731: 8 total, 8 up, 8 in 2024-03-12T03:30:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-12T03:30:07.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:30:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:07 smithi094 ceph-mon[40484]: osdmap e732: 8 total, 8 up, 8 in 2024-03-12T03:30:07.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:30:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:30:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:07 smithi094 ceph-mon[29291]: osdmap e732: 8 total, 8 up, 8 in 2024-03-12T03:30:07.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:30:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_count","val": "2"}]': finished 2024-03-12T03:30:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:07 smithi151 ceph-mon[32923]: osdmap e732: 8 total, 8 up, 8 in 2024-03-12T03:30:07.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-12T03:30:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:08 smithi151 ceph-mon[32923]: pgmap v1138: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-12T03:30:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:30:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:08 smithi151 ceph-mon[32923]: osdmap e733: 8 total, 8 up, 8 in 2024-03-12T03:30:08.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:30:08.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:30:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:30:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:08 smithi094 ceph-mon[29291]: pgmap v1138: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-12T03:30:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:30:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:08 smithi094 ceph-mon[29291]: osdmap e733: 8 total, 8 up, 8 in 2024-03-12T03:30:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:30:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:08 smithi094 ceph-mon[40484]: pgmap v1138: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-12T03:30:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_period","val": "600"}]': finished 2024-03-12T03:30:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:08 smithi094 ceph-mon[40484]: osdmap e733: 8 total, 8 up, 8 in 2024-03-12T03:30:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-12T03:30:09.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:09.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:09 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:30:09.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:30:09.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:09 smithi094 ceph-mon[40484]: osdmap e734: 8 total, 8 up, 8 in 2024-03-12T03:30:09.962 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:30:09.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:09.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:09 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:30:09.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:30:09.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:09 smithi094 ceph-mon[29291]: osdmap e734: 8 total, 8 up, 8 in 2024-03-12T03:30:09.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:30:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:09 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-12T03:30:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-12T03:30:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:09 smithi151 ceph-mon[32923]: osdmap e734: 8 total, 8 up, 8 in 2024-03-12T03:30:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-12T03:30:10.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:30:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:30:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:30:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:10 smithi094 ceph-mon[40484]: pgmap v1141: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-12T03:30:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-12T03:30:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:10 smithi094 ceph-mon[40484]: osdmap e735: 8 total, 8 up, 8 in 2024-03-12T03:30:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-12T03:30:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:10 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "target_max_objects","val": "1"}]': finished 2024-03-12T03:30:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:10 smithi094 ceph-mon[40484]: osdmap e736: 8 total, 8 up, 8 in 2024-03-12T03:30:10.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:10 smithi094 ceph-mon[29291]: pgmap v1141: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-12T03:30:10.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-12T03:30:10.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:10 smithi094 ceph-mon[29291]: osdmap e735: 8 total, 8 up, 8 in 2024-03-12T03:30:10.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-12T03:30:10.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:10 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "target_max_objects","val": "1"}]': finished 2024-03-12T03:30:10.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:10 smithi094 ceph-mon[29291]: osdmap e736: 8 total, 8 up, 8 in 2024-03-12T03:30:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:10 smithi151 ceph-mon[32923]: pgmap v1141: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-12T03:30:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-12T03:30:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:10 smithi151 ceph-mon[32923]: osdmap e735: 8 total, 8 up, 8 in 2024-03-12T03:30:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-12T03:30:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:10 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-146","var": "target_max_objects","val": "1"}]': finished 2024-03-12T03:30:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:10 smithi151 ceph-mon[32923]: osdmap e736: 8 total, 8 up, 8 in 2024-03-12T03:30:11.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:11 smithi094 ceph-mon[29291]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-12T03:30:11.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:11 smithi094 ceph-mon[40484]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-12T03:30:11.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:11 smithi151 ceph-mon[32923]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-12T03:30:12.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:12 smithi094 ceph-mon[29291]: pgmap v1144: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-12T03:30:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:12 smithi094 ceph-mon[40484]: pgmap v1144: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-12T03:30:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:12 smithi151 ceph-mon[32923]: pgmap v1144: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-12T03:30:13.714 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:30:13 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=sqlstore.transactions t=2024-03-12T03:30:13.344083014Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-12T03:30:14.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:14 smithi094 ceph-mon[29291]: pgmap v1145: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-12T03:30:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:14 smithi094 ceph-mon[40484]: pgmap v1145: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-12T03:30:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:14 smithi151 ceph-mon[32923]: pgmap v1145: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-12T03:30:16.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:16 smithi094 ceph-mon[29291]: pgmap v1146: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 808 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:30:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:16 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:16 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:16 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:16 smithi094 ceph-mon[40484]: pgmap v1146: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 808 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:30:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:16 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:16 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:16 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:16 smithi151 ceph-mon[32923]: pgmap v1146: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 808 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:30:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:16 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:16 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:16 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:17.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:17.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:17 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:17.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:17 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:17.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:17.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:17.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:17 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:18 smithi151 ceph-mon[32923]: pgmap v1147: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-12T03:30:18.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:30:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:30:18.861 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:18 smithi094 ceph-mon[40484]: pgmap v1147: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-12T03:30:18.861 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:18 smithi094 ceph-mon[29291]: pgmap v1147: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-12T03:30:19.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:20.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:30:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:30:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:30:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:20 smithi094 ceph-mon[29291]: pgmap v1148: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:30:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:20 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:30:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:20 smithi094 ceph-mon[40484]: pgmap v1148: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:30:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:20 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:30:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:20 smithi151 ceph-mon[32923]: pgmap v1148: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:30:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:20 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:30:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:22.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:30:22.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:21 smithi094 ceph-mon[40484]: osdmap e737: 8 total, 8 up, 8 in 2024-03-12T03:30:22.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146"}]: dispatch 2024-03-12T03:30:22.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:21 smithi094 ceph-mon[40484]: pgmap v1150: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:30:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:30:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:21 smithi094 ceph-mon[29291]: osdmap e737: 8 total, 8 up, 8 in 2024-03-12T03:30:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146"}]: dispatch 2024-03-12T03:30:22.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:21 smithi094 ceph-mon[29291]: pgmap v1150: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:30:22.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:30:22.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:21 smithi151 ceph-mon[32923]: osdmap e737: 8 total, 8 up, 8 in 2024-03-12T03:30:22.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146"}]: dispatch 2024-03-12T03:30:22.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:21 smithi151 ceph-mon[32923]: pgmap v1150: 268 pgs: 268 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-12T03:30:23.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146"}]': finished 2024-03-12T03:30:23.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:22 smithi094 ceph-mon[40484]: osdmap e738: 8 total, 8 up, 8 in 2024-03-12T03:30:23.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:23.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146"}]: dispatch 2024-03-12T03:30:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146"}]': finished 2024-03-12T03:30:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:22 smithi094 ceph-mon[29291]: osdmap e738: 8 total, 8 up, 8 in 2024-03-12T03:30:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:23.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146"}]: dispatch 2024-03-12T03:30:23.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146"}]': finished 2024-03-12T03:30:23.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:22 smithi151 ceph-mon[32923]: osdmap e738: 8 total, 8 up, 8 in 2024-03-12T03:30:23.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:23.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-146"}]: dispatch 2024-03-12T03:30:24.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:23 smithi094 ceph-mon[29291]: osdmap e739: 8 total, 8 up, 8 in 2024-03-12T03:30:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:23 smithi094 ceph-mon[29291]: pgmap v1153: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:30:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:23 smithi094 ceph-mon[29291]: osdmap e740: 8 total, 8 up, 8 in 2024-03-12T03:30:24.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:23 smithi094 ceph-mon[40484]: osdmap e739: 8 total, 8 up, 8 in 2024-03-12T03:30:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:23 smithi094 ceph-mon[40484]: pgmap v1153: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:30:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:23 smithi094 ceph-mon[40484]: osdmap e740: 8 total, 8 up, 8 in 2024-03-12T03:30:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:23 smithi151 ceph-mon[32923]: osdmap e739: 8 total, 8 up, 8 in 2024-03-12T03:30:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:23 smithi151 ceph-mon[32923]: pgmap v1153: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:30:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:23 smithi151 ceph-mon[32923]: osdmap e740: 8 total, 8 up, 8 in 2024-03-12T03:30:24.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:25.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:24 smithi094 ceph-mon[29291]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-12T03:30:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:24 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:24 smithi094 ceph-mon[29291]: osdmap e741: 8 total, 8 up, 8 in 2024-03-12T03:30:25.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:24 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:30:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:24 smithi094 ceph-mon[40484]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-12T03:30:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:24 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:24 smithi094 ceph-mon[40484]: osdmap e741: 8 total, 8 up, 8 in 2024-03-12T03:30:25.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:24 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:30:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:24 smithi151 ceph-mon[32923]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-12T03:30:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:24 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:24 smithi151 ceph-mon[32923]: osdmap e741: 8 total, 8 up, 8 in 2024-03-12T03:30:25.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:24 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-12T03:30:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:25 smithi094 ceph-mon[29291]: pgmap v1156: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:30:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:30:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:25 smithi094 ceph-mon[29291]: osdmap e742: 8 total, 8 up, 8 in 2024-03-12T03:30:26.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148"}]: dispatch 2024-03-12T03:30:26.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:25 smithi094 ceph-mon[40484]: pgmap v1156: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:30:26.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:30:26.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:25 smithi094 ceph-mon[40484]: osdmap e742: 8 total, 8 up, 8 in 2024-03-12T03:30:26.461 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148"}]: dispatch 2024-03-12T03:30:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:25 smithi151 ceph-mon[32923]: pgmap v1156: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:30:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-12T03:30:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:25 smithi151 ceph-mon[32923]: osdmap e742: 8 total, 8 up, 8 in 2024-03-12T03:30:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148"}]: dispatch 2024-03-12T03:30:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148"}]': finished 2024-03-12T03:30:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:27 smithi094 ceph-mon[40484]: osdmap e743: 8 total, 8 up, 8 in 2024-03-12T03:30:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148"}]: dispatch 2024-03-12T03:30:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:27 smithi094 ceph-mon[40484]: pgmap v1159: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:30:28.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148"}]': finished 2024-03-12T03:30:28.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:27 smithi094 ceph-mon[29291]: osdmap e743: 8 total, 8 up, 8 in 2024-03-12T03:30:28.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:28.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148"}]: dispatch 2024-03-12T03:30:28.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:27 smithi094 ceph-mon[29291]: pgmap v1159: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:30:28.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148"}]': finished 2024-03-12T03:30:28.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:27 smithi151 ceph-mon[32923]: osdmap e743: 8 total, 8 up, 8 in 2024-03-12T03:30:28.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:28.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-148"}]: dispatch 2024-03-12T03:30:28.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:27 smithi151 ceph-mon[32923]: pgmap v1159: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:30:28.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:30:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:30:29.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:28 smithi094 ceph-mon[29291]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:30:29.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:28 smithi094 ceph-mon[29291]: osdmap e744: 8 total, 8 up, 8 in 2024-03-12T03:30:29.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:28 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:28 smithi094 ceph-mon[40484]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:30:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:28 smithi094 ceph-mon[40484]: osdmap e744: 8 total, 8 up, 8 in 2024-03-12T03:30:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:28 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:29.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:28 smithi151 ceph-mon[32923]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:30:29.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:28 smithi151 ceph-mon[32923]: osdmap e744: 8 total, 8 up, 8 in 2024-03-12T03:30:29.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:28 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:30.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:29 smithi094 ceph-mon[29291]: osdmap e745: 8 total, 8 up, 8 in 2024-03-12T03:30:30.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:30.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:29 smithi094 ceph-mon[29291]: pgmap v1162: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:30:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:29 smithi094 ceph-mon[40484]: osdmap e745: 8 total, 8 up, 8 in 2024-03-12T03:30:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:29 smithi094 ceph-mon[40484]: pgmap v1162: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:30:30.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:29 smithi151 ceph-mon[32923]: osdmap e745: 8 total, 8 up, 8 in 2024-03-12T03:30:30.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:30.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:29 smithi151 ceph-mon[32923]: pgmap v1162: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:30:30.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:30:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:30:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:30:30.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:30 smithi094 ceph-mon[29291]: osdmap e746: 8 total, 8 up, 8 in 2024-03-12T03:30:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:30 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-150"}]: dispatch 2024-03-12T03:30:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:30 smithi094 ceph-mon[40484]: osdmap e746: 8 total, 8 up, 8 in 2024-03-12T03:30:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:30 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-150"}]: dispatch 2024-03-12T03:30:31.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:31.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:30 smithi151 ceph-mon[32923]: osdmap e746: 8 total, 8 up, 8 in 2024-03-12T03:30:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:30 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-150"}]: dispatch 2024-03-12T03:30:32.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:31 smithi094 ceph-mon[29291]: osdmap e747: 8 total, 8 up, 8 in 2024-03-12T03:30:32.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:31 smithi094 ceph-mon[29291]: pgmap v1165: 236 pgs: 236 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:30:32.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:31 smithi094 ceph-mon[40484]: osdmap e747: 8 total, 8 up, 8 in 2024-03-12T03:30:32.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:31 smithi094 ceph-mon[40484]: pgmap v1165: 236 pgs: 236 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:30:32.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:31 smithi151 ceph-mon[32923]: osdmap e747: 8 total, 8 up, 8 in 2024-03-12T03:30:32.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:31 smithi151 ceph-mon[32923]: pgmap v1165: 236 pgs: 236 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-12T03:30:33.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:32 smithi094 ceph-mon[29291]: osdmap e748: 8 total, 8 up, 8 in 2024-03-12T03:30:33.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:32 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:33.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:32 smithi094 ceph-mon[40484]: osdmap e748: 8 total, 8 up, 8 in 2024-03-12T03:30:33.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:32 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:33.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:32 smithi151 ceph-mon[32923]: osdmap e748: 8 total, 8 up, 8 in 2024-03-12T03:30:33.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:32 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:34.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:34.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:33 smithi094 ceph-mon[29291]: osdmap e749: 8 total, 8 up, 8 in 2024-03-12T03:30:34.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:34.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:33 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-152"}]: dispatch 2024-03-12T03:30:34.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:33 smithi094 ceph-mon[29291]: pgmap v1168: 268 pgs: 14 creating+peering, 14 unknown, 240 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-12T03:30:34.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:33 smithi094 ceph-mon[29291]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:30:34.218 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:34.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:33 smithi094 ceph-mon[40484]: osdmap e749: 8 total, 8 up, 8 in 2024-03-12T03:30:34.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:34.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:33 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-152"}]: dispatch 2024-03-12T03:30:34.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:33 smithi094 ceph-mon[40484]: pgmap v1168: 268 pgs: 14 creating+peering, 14 unknown, 240 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-12T03:30:34.219 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:33 smithi094 ceph-mon[40484]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:30:34.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:34.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:33 smithi151 ceph-mon[32923]: osdmap e749: 8 total, 8 up, 8 in 2024-03-12T03:30:34.219 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:34.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:33 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-152"}]: dispatch 2024-03-12T03:30:34.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:33 smithi151 ceph-mon[32923]: pgmap v1168: 268 pgs: 14 creating+peering, 14 unknown, 240 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-12T03:30:34.220 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:33 smithi151 ceph-mon[32923]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:30:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:34 smithi094 ceph-mon[29291]: osdmap e750: 8 total, 8 up, 8 in 2024-03-12T03:30:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:34 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:30:35.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:34 smithi094 ceph-mon[40484]: osdmap e750: 8 total, 8 up, 8 in 2024-03-12T03:30:35.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:34 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:35.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:30:35.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:34 smithi151 ceph-mon[32923]: osdmap e750: 8 total, 8 up, 8 in 2024-03-12T03:30:35.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:34 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:30:35.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:30:36.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:35 smithi094 ceph-mon[40484]: osdmap e751: 8 total, 8 up, 8 in 2024-03-12T03:30:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:35 smithi094 ceph-mon[40484]: pgmap v1171: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-12T03:30:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:35 smithi094 ceph-mon[40484]: osdmap e752: 8 total, 8 up, 8 in 2024-03-12T03:30:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:35 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-111","var": "dedup_tier","val": "test-rados-api-smithi094-78452-154"}]: dispatch 2024-03-12T03:30:36.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:35 smithi094 ceph-mon[29291]: osdmap e751: 8 total, 8 up, 8 in 2024-03-12T03:30:36.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:36.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:35 smithi094 ceph-mon[29291]: pgmap v1171: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-12T03:30:36.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:36.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:35 smithi094 ceph-mon[29291]: osdmap e752: 8 total, 8 up, 8 in 2024-03-12T03:30:36.211 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:35 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-111","var": "dedup_tier","val": "test-rados-api-smithi094-78452-154"}]: dispatch 2024-03-12T03:30:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:35 smithi151 ceph-mon[32923]: osdmap e751: 8 total, 8 up, 8 in 2024-03-12T03:30:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:30:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:35 smithi151 ceph-mon[32923]: pgmap v1171: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-12T03:30:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi094-78452-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-12T03:30:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:35 smithi151 ceph-mon[32923]: osdmap e752: 8 total, 8 up, 8 in 2024-03-12T03:30:36.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:35 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi094-78452-111","var": "dedup_tier","val": "test-rados-api-smithi094-78452-154"}]: dispatch 2024-03-12T03:30:37.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:37.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:36 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-154"}]: dispatch 2024-03-12T03:30:37.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:36 smithi094 ceph-mon[29291]: osdmap e753: 8 total, 8 up, 8 in 2024-03-12T03:30:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:36 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-154"}]: dispatch 2024-03-12T03:30:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:36 smithi094 ceph-mon[40484]: osdmap e753: 8 total, 8 up, 8 in 2024-03-12T03:30:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:36 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi094-78452-111", "tierpool": "test-rados-api-smithi094-78452-154"}]: dispatch 2024-03-12T03:30:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:36 smithi151 ceph-mon[32923]: osdmap e753: 8 total, 8 up, 8 in 2024-03-12T03:30:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:37 smithi094 ceph-mon[40484]: pgmap v1174: 236 pgs: 236 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:30:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:37 smithi094 ceph-mon[40484]: osdmap e754: 8 total, 8 up, 8 in 2024-03-12T03:30:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:37 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:37 smithi094 ceph-mon[29291]: pgmap v1174: 236 pgs: 236 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:30:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:37 smithi094 ceph-mon[29291]: osdmap e754: 8 total, 8 up, 8 in 2024-03-12T03:30:38.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:37 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:38.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:37 smithi151 ceph-mon[32923]: pgmap v1174: 236 pgs: 236 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:30:38.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:37 smithi151 ceph-mon[32923]: osdmap e754: 8 total, 8 up, 8 in 2024-03-12T03:30:38.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:37 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:38.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:30:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:30:39.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:30:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:39 smithi094 ceph-mon[40484]: osdmap e755: 8 total, 8 up, 8 in 2024-03-12T03:30:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:39 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:30:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:39 smithi094 ceph-mon[29291]: osdmap e755: 8 total, 8 up, 8 in 2024-03-12T03:30:39.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:39 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:30:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:39 smithi151 ceph-mon[32923]: osdmap e755: 8 total, 8 up, 8 in 2024-03-12T03:30:39.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:39 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-111"}]: dispatch 2024-03-12T03:30:39.814 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7845 ms) 2024-03-12T03:30:39.814 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-12T03:30:39.814 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14314 ms) 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8037 ms) 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13072 ms) 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7597 ms) 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8043 ms) 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: ok, hit_set contains 322:602f83fe:::foo:head 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8311 ms) 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710214155,0 2024-03-12T03:30:39.815 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: first is 1710214155 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710214155,0 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710214155,0 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710214155,1710214157,1710214158,0 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710214155,1710214157,1710214158,0 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710214155,1710214157,1710214158,0 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710214155,1710214157,1710214158,1710214160,1710214161,0 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710214155,1710214157,1710214158,1710214160,1710214161,0 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710214155,1710214157,1710214158,1710214160,1710214161,0 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710214155,1710214157,1710214158,1710214160,1710214161,1710214163,1710214164,0 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710214155,1710214157,1710214158,1710214160,1710214161,1710214163,1710214164,0 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710214155,1710214157,1710214158,1710214160,1710214161,1710214163,1710214164,0 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: got ls 1710214158,1710214160,1710214161,1710214163,1710214164,1710214166,1710214167,0 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: first now 1710214158, trimmed 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20662 ms) 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-12T03:30:39.816 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: foo0 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14063 ms) 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18064 ms) 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22285 ms) 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5015 ms) 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3018 ms) 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3023 ms) 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3018 ms) 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (232022 ms total) 2024-03-12T03:30:39.817 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: 2024-03-12T03:30:39.818 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-12T03:30:39.818 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (863006 ms total) 2024-03-12T03:30:39.818 INFO:tasks.workunit.client.0.smithi094.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-12T03:30:39.820 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.820 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78054 2024-03-12T03:30:39.820 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78054 2024-03-12T03:30:39.820 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78279 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78279 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78784 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78784 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78554 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78554 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78386 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78386 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78141 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78141 2024-03-12T03:30:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78476 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78476 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79023 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 79023 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79068 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 79068 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78159 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78159 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78838 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78838 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78686 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78686 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78029 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78029 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78093 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78093 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78709 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78709 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78020 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78020 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78040 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78040 2024-03-12T03:30:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.824 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79200 2024-03-12T03:30:39.824 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 79200 2024-03-12T03:30:39.824 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.824 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78300 2024-03-12T03:30:39.824 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78300 2024-03-12T03:30:39.824 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:30:39.824 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79285 2024-03-12T03:30:39.824 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 79285 2024-03-12T03:30:40.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:30:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:30:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:30:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:40 smithi094 ceph-mon[29291]: pgmap v1177: 228 pgs: 228 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:30:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:40 smithi094 ceph-mon[29291]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:30:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:30:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:40 smithi094 ceph-mon[29291]: osdmap e756: 8 total, 8 up, 8 in 2024-03-12T03:30:40.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:40 smithi094 ceph-mon[40484]: pgmap v1177: 228 pgs: 228 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:30:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:40 smithi094 ceph-mon[40484]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:30:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:30:40.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:40 smithi094 ceph-mon[40484]: osdmap e756: 8 total, 8 up, 8 in 2024-03-12T03:30:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:40 smithi151 ceph-mon[32923]: pgmap v1177: 228 pgs: 228 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:30:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:40 smithi151 ceph-mon[32923]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:30:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/354411895' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi094-78452-111"}]': finished 2024-03-12T03:30:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:40 smithi151 ceph-mon[32923]: osdmap e756: 8 total, 8 up, 8 in 2024-03-12T03:30:42.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:42 smithi094 ceph-mon[29291]: pgmap v1179: 228 pgs: 228 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:30:42.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:42 smithi094 ceph-mon[40484]: pgmap v1179: 228 pgs: 228 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:30:42.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:42 smithi151 ceph-mon[32923]: pgmap v1179: 228 pgs: 228 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:30:44.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:44 smithi094 ceph-mon[29291]: pgmap v1180: 228 pgs: 228 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:30:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:44 smithi094 ceph-mon[40484]: pgmap v1180: 228 pgs: 228 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:30:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:44 smithi151 ceph-mon[32923]: pgmap v1180: 228 pgs: 228 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:30:46.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:46 smithi094 ceph-mon[29291]: pgmap v1181: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 728 B/s rd, 0 op/s 2024-03-12T03:30:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:46 smithi094 ceph-mon[40484]: pgmap v1181: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 728 B/s rd, 0 op/s 2024-03-12T03:30:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:46 smithi151 ceph-mon[32923]: pgmap v1181: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 728 B/s rd, 0 op/s 2024-03-12T03:30:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:48 smithi094 ceph-mon[29291]: pgmap v1182: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-03-12T03:30:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:48 smithi094 ceph-mon[40484]: pgmap v1182: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-03-12T03:30:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:48 smithi151 ceph-mon[32923]: pgmap v1182: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-03-12T03:30:48.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:30:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:30:49.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:50.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:50 smithi094 ceph-mon[40484]: pgmap v1183: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:30:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:30:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:50 smithi094 ceph-mon[29291]: pgmap v1183: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:30:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:30:50.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:30:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:30:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:30:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:50 smithi151 ceph-mon[32923]: pgmap v1183: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:30:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:30:52.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:52 smithi094 ceph-mon[29291]: pgmap v1184: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-03-12T03:30:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:52 smithi094 ceph-mon[40484]: pgmap v1184: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-03-12T03:30:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:52 smithi151 ceph-mon[32923]: pgmap v1184: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-03-12T03:30:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:54 smithi094 ceph-mon[29291]: pgmap v1185: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:30:54.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:54 smithi094 ceph-mon[40484]: pgmap v1185: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:30:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:54 smithi151 ceph-mon[32923]: pgmap v1185: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:30:56.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:56 smithi094 ceph-mon[29291]: pgmap v1186: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:30:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:56 smithi094 ceph-mon[40484]: pgmap v1186: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:30:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:56 smithi151 ceph-mon[32923]: pgmap v1186: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:30:58.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:58 smithi094 ceph-mon[29291]: pgmap v1187: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:30:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:58 smithi094 ceph-mon[40484]: pgmap v1187: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:30:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:58 smithi151 ceph-mon[32923]: pgmap v1187: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:30:58.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:30:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:30:59.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:30:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:30:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:30:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:30:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:00.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:00 smithi094 ceph-mon[29291]: pgmap v1188: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:00 smithi094 ceph-mon[40484]: pgmap v1188: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:00.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:31:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:31:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:31:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:00 smithi151 ceph-mon[32923]: pgmap v1188: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:02.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:02 smithi094 ceph-mon[29291]: pgmap v1189: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:02 smithi094 ceph-mon[40484]: pgmap v1189: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:02 smithi151 ceph-mon[32923]: pgmap v1189: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:04.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:04 smithi094 ceph-mon[40484]: pgmap v1190: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:04.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:04 smithi094 ceph-mon[29291]: pgmap v1190: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:04 smithi151 ceph-mon[32923]: pgmap v1190: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:05.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:31:05.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:31:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:06 smithi094 ceph-mon[40484]: pgmap v1191: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:06.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:06 smithi094 ceph-mon[29291]: pgmap v1191: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:06 smithi151 ceph-mon[32923]: pgmap v1191: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:08.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:08 smithi094 ceph-mon[29291]: pgmap v1192: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:08 smithi094 ceph-mon[40484]: pgmap v1192: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:08 smithi151 ceph-mon[32923]: pgmap v1192: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:08.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:31:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:31:09.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:09.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:10.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:10 smithi094 ceph-mon[29291]: pgmap v1193: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:10 smithi094 ceph-mon[40484]: pgmap v1193: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:10.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:31:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:31:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:31:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:10 smithi151 ceph-mon[32923]: pgmap v1193: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:12.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:12 smithi094 ceph-mon[29291]: pgmap v1194: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:12.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:12 smithi094 ceph-mon[40484]: pgmap v1194: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:12 smithi151 ceph-mon[32923]: pgmap v1194: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:14 smithi094 ceph-mon[29291]: pgmap v1195: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:14 smithi094 ceph-mon[40484]: pgmap v1195: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:14 smithi151 ceph-mon[32923]: pgmap v1195: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:16.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:16 smithi094 ceph-mon[29291]: pgmap v1196: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:16.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:16 smithi094 ceph-mon[40484]: pgmap v1196: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:16.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:16 smithi151 ceph-mon[32923]: pgmap v1196: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:17.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:17 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:17.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:17 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:17.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:17 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:18 smithi094 ceph-mon[29291]: pgmap v1197: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:18 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:18 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:18.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:18 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:31:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:18 smithi094 ceph-mon[40484]: pgmap v1197: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:18 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:18 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:18 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:31:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:18 smithi151 ceph-mon[32923]: pgmap v1197: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:18 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:18 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:18.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:18 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:31:18.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:31:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:31:19.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:20.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:20 smithi094 ceph-mon[29291]: pgmap v1198: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:31:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:20 smithi094 ceph-mon[40484]: pgmap v1198: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:31:20.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:31:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:31:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:31:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:20 smithi151 ceph-mon[32923]: pgmap v1198: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:31:22.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:22 smithi094 ceph-mon[29291]: pgmap v1199: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:22.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:22 smithi094 ceph-mon[40484]: pgmap v1199: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:22 smithi151 ceph-mon[32923]: pgmap v1199: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:24.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:24 smithi094 ceph-mon[29291]: pgmap v1200: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:24.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:24 smithi094 ceph-mon[40484]: pgmap v1200: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:24 smithi151 ceph-mon[32923]: pgmap v1200: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:26.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:26 smithi094 ceph-mon[29291]: pgmap v1201: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:26 smithi094 ceph-mon[40484]: pgmap v1201: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:26 smithi151 ceph-mon[32923]: pgmap v1201: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:28.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:28 smithi094 ceph-mon[29291]: pgmap v1202: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:28 smithi094 ceph-mon[40484]: pgmap v1202: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:28 smithi151 ceph-mon[32923]: pgmap v1202: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:28.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:31:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:31:29.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:30.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:30 smithi094 ceph-mon[29291]: pgmap v1203: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:30 smithi094 ceph-mon[40484]: pgmap v1203: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:30.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:31:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:31:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:31:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:30 smithi151 ceph-mon[32923]: pgmap v1203: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:32.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:32 smithi094 ceph-mon[29291]: pgmap v1204: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:32 smithi094 ceph-mon[40484]: pgmap v1204: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:32 smithi151 ceph-mon[32923]: pgmap v1204: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:34.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:34 smithi094 ceph-mon[29291]: pgmap v1205: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:34 smithi094 ceph-mon[40484]: pgmap v1205: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:34 smithi151 ceph-mon[32923]: pgmap v1205: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:35.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:31:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:31:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:31:36.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:36 smithi094 ceph-mon[29291]: pgmap v1206: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:36 smithi094 ceph-mon[40484]: pgmap v1206: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:36 smithi151 ceph-mon[32923]: pgmap v1206: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:38.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:38 smithi094 ceph-mon[29291]: pgmap v1207: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:38 smithi094 ceph-mon[40484]: pgmap v1207: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:38.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:38 smithi151 ceph-mon[32923]: pgmap v1207: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:38.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:31:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:31:39.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:40 smithi094 ceph-mon[29291]: pgmap v1208: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:40.806 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:40 smithi094 ceph-mon[40484]: pgmap v1208: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:40.806 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:31:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:31:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:31:40.807 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:40 smithi151 ceph-mon[32923]: pgmap v1208: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:42.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:42 smithi094 ceph-mon[29291]: pgmap v1209: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:42.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:42 smithi094 ceph-mon[40484]: pgmap v1209: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:42.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:42 smithi151 ceph-mon[32923]: pgmap v1209: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:44.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:44 smithi094 ceph-mon[40484]: pgmap v1210: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:44 smithi094 ceph-mon[29291]: pgmap v1210: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:44 smithi151 ceph-mon[32923]: pgmap v1210: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:46.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:46 smithi094 ceph-mon[29291]: pgmap v1211: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:46 smithi094 ceph-mon[40484]: pgmap v1211: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:46 smithi151 ceph-mon[32923]: pgmap v1211: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:48.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:48 smithi094 ceph-mon[29291]: pgmap v1212: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:48 smithi094 ceph-mon[40484]: pgmap v1212: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:48 smithi151 ceph-mon[32923]: pgmap v1212: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:48.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:31:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:31:49.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:50.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:50 smithi094 ceph-mon[40484]: pgmap v1213: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:50.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:31:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:50 smithi094 ceph-mon[29291]: pgmap v1213: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:31:50.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:31:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:31:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:31:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:50 smithi151 ceph-mon[32923]: pgmap v1213: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:31:52.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:52 smithi094 ceph-mon[29291]: pgmap v1214: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:52 smithi094 ceph-mon[40484]: pgmap v1214: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:52 smithi151 ceph-mon[32923]: pgmap v1214: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:31:54.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:54 smithi094 ceph-mon[29291]: pgmap v1215: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:54.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:54 smithi094 ceph-mon[40484]: pgmap v1215: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:54 smithi151 ceph-mon[32923]: pgmap v1215: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:31:56.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:56 smithi094 ceph-mon[29291]: pgmap v1216: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:56 smithi094 ceph-mon[40484]: pgmap v1216: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:56 smithi151 ceph-mon[32923]: pgmap v1216: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:58.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:58 smithi094 ceph-mon[29291]: pgmap v1217: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:58 smithi094 ceph-mon[40484]: pgmap v1217: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:58 smithi151 ceph-mon[32923]: pgmap v1217: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:31:58.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:31:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:31:59.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:31:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:59.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:31:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:31:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:31:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:00.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:00 smithi094 ceph-mon[29291]: pgmap v1218: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:00 smithi094 ceph-mon[40484]: pgmap v1218: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:00.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:32:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:32:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:32:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:00 smithi151 ceph-mon[32923]: pgmap v1218: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:02.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:02 smithi094 ceph-mon[29291]: pgmap v1219: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:02 smithi094 ceph-mon[40484]: pgmap v1219: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:02 smithi151 ceph-mon[32923]: pgmap v1219: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:04 smithi151 ceph-mon[32923]: pgmap v1220: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:04 smithi094 ceph-mon[29291]: pgmap v1220: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:04 smithi094 ceph-mon[40484]: pgmap v1220: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:05.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:32:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:32:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:32:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:06 smithi094 ceph-mon[29291]: pgmap v1221: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:06 smithi094 ceph-mon[40484]: pgmap v1221: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:06 smithi151 ceph-mon[32923]: pgmap v1221: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:08 smithi094 ceph-mon[29291]: pgmap v1222: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:08 smithi094 ceph-mon[40484]: pgmap v1222: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:08 smithi151 ceph-mon[32923]: pgmap v1222: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:08.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:32:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:32:09.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:10.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:10 smithi094 ceph-mon[29291]: pgmap v1223: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:10 smithi094 ceph-mon[40484]: pgmap v1223: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:32:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:32:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:32:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:10 smithi151 ceph-mon[32923]: pgmap v1223: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:12.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:12 smithi094 ceph-mon[29291]: pgmap v1224: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:12 smithi094 ceph-mon[40484]: pgmap v1224: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:12 smithi151 ceph-mon[32923]: pgmap v1224: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:14.766 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:14 smithi151 ceph-mon[32923]: pgmap v1225: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:14.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:14 smithi094 ceph-mon[29291]: pgmap v1225: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:14 smithi094 ceph-mon[40484]: pgmap v1225: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:16.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:16 smithi094 ceph-mon[29291]: pgmap v1226: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:16 smithi094 ceph-mon[40484]: pgmap v1226: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:16 smithi151 ceph-mon[32923]: pgmap v1226: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:18.762 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:18 smithi094 ceph-mon[40484]: pgmap v1227: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:18.762 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:18 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:18.762 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:18 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:18.762 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:18 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:18.762 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:18 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:32:18.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:18 smithi094 ceph-mon[29291]: pgmap v1227: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:18.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:18 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:18.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:18 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:18.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:18 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:18.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:18 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:32:18.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:32:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:32:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:18 smithi151 ceph-mon[32923]: pgmap v1227: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:18 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:18 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:18 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:18 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:32:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:20.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:20 smithi094 ceph-mon[29291]: pgmap v1228: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:20.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:32:20.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:32:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:32:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:32:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:20 smithi094 ceph-mon[40484]: pgmap v1228: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:32:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:20 smithi151 ceph-mon[32923]: pgmap v1228: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:32:22.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:22 smithi094 ceph-mon[29291]: pgmap v1229: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:22 smithi094 ceph-mon[40484]: pgmap v1229: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:22 smithi151 ceph-mon[32923]: pgmap v1229: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:24.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:24 smithi094 ceph-mon[29291]: pgmap v1230: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:24.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:24 smithi094 ceph-mon[40484]: pgmap v1230: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:24 smithi151 ceph-mon[32923]: pgmap v1230: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:26.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:26 smithi094 ceph-mon[40484]: pgmap v1231: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:26.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:26 smithi094 ceph-mon[29291]: pgmap v1231: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:26 smithi151 ceph-mon[32923]: pgmap v1231: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:28 smithi094 ceph-mon[29291]: pgmap v1232: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:28 smithi094 ceph-mon[40484]: pgmap v1232: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:28 smithi151 ceph-mon[32923]: pgmap v1232: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:28.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:32:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:32:29.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:30.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:30 smithi094 ceph-mon[29291]: pgmap v1233: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:30 smithi094 ceph-mon[40484]: pgmap v1233: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:30.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:32:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:32:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:32:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:30 smithi151 ceph-mon[32923]: pgmap v1233: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:32.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:32 smithi094 ceph-mon[29291]: pgmap v1234: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:32.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:32 smithi094 ceph-mon[40484]: pgmap v1234: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:32 smithi151 ceph-mon[32923]: pgmap v1234: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:34.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:34 smithi094 ceph-mon[29291]: pgmap v1235: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:34 smithi094 ceph-mon[40484]: pgmap v1235: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:34 smithi151 ceph-mon[32923]: pgmap v1235: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:35.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:32:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:32:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:32:36.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:36 smithi094 ceph-mon[29291]: pgmap v1236: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:36 smithi094 ceph-mon[40484]: pgmap v1236: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:36 smithi151 ceph-mon[32923]: pgmap v1236: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:38 smithi094 ceph-mon[29291]: pgmap v1237: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:38 smithi094 ceph-mon[40484]: pgmap v1237: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:38 smithi151 ceph-mon[32923]: pgmap v1237: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:38.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:32:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:32:39.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:40.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:40 smithi094 ceph-mon[29291]: pgmap v1238: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:40 smithi094 ceph-mon[40484]: pgmap v1238: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:32:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:32:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:32:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:40 smithi151 ceph-mon[32923]: pgmap v1238: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:42 smithi094 ceph-mon[29291]: pgmap v1239: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:42 smithi094 ceph-mon[40484]: pgmap v1239: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:42 smithi151 ceph-mon[32923]: pgmap v1239: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:44.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:44 smithi094 ceph-mon[40484]: pgmap v1240: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:44 smithi094 ceph-mon[29291]: pgmap v1240: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:44 smithi151 ceph-mon[32923]: pgmap v1240: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:46.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:46 smithi094 ceph-mon[29291]: pgmap v1241: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:46 smithi094 ceph-mon[40484]: pgmap v1241: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:46 smithi151 ceph-mon[32923]: pgmap v1241: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:48.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:48 smithi094 ceph-mon[29291]: pgmap v1242: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:48.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:48 smithi094 ceph-mon[40484]: pgmap v1242: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:48 smithi151 ceph-mon[32923]: pgmap v1242: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:32:48.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:32:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:32:49.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:49.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:49.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:50.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:50 smithi094 ceph-mon[40484]: pgmap v1243: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:32:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:50 smithi094 ceph-mon[29291]: pgmap v1243: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:32:50.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:32:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:32:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:32:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:50 smithi151 ceph-mon[32923]: pgmap v1243: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:32:52.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:52 smithi094 ceph-mon[29291]: pgmap v1244: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:52 smithi094 ceph-mon[40484]: pgmap v1244: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:52.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:52 smithi151 ceph-mon[32923]: pgmap v1244: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:54.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:54 smithi094 ceph-mon[29291]: pgmap v1245: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:54.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:54 smithi094 ceph-mon[40484]: pgmap v1245: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:54 smithi151 ceph-mon[32923]: pgmap v1245: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:32:56.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:56 smithi094 ceph-mon[29291]: pgmap v1246: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:56.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:56 smithi094 ceph-mon[40484]: pgmap v1246: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:56.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:56 smithi151 ceph-mon[32923]: pgmap v1246: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:58 smithi094 ceph-mon[40484]: pgmap v1247: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:58.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:58 smithi094 ceph-mon[29291]: pgmap v1247: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:58 smithi151 ceph-mon[32923]: pgmap v1247: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:32:58.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:32:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:32:59.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:32:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:59.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:32:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:32:59.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:32:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:00 smithi094 ceph-mon[29291]: pgmap v1248: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:00 smithi094 ceph-mon[40484]: pgmap v1248: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:00.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:33:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:33:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:33:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:00 smithi151 ceph-mon[32923]: pgmap v1248: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:02.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:02 smithi094 ceph-mon[29291]: pgmap v1249: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:02 smithi094 ceph-mon[40484]: pgmap v1249: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:02 smithi151 ceph-mon[32923]: pgmap v1249: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:04 smithi094 ceph-mon[29291]: pgmap v1250: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:04 smithi094 ceph-mon[40484]: pgmap v1250: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:04 smithi151 ceph-mon[32923]: pgmap v1250: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:05.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:33:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:33:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:33:06.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:06 smithi094 ceph-mon[40484]: pgmap v1251: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:06 smithi094 ceph-mon[29291]: pgmap v1251: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:06.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:06 smithi151 ceph-mon[32923]: pgmap v1251: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:08 smithi094 ceph-mon[29291]: pgmap v1252: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:08 smithi094 ceph-mon[40484]: pgmap v1252: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:08 smithi151 ceph-mon[32923]: pgmap v1252: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:08.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:33:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:33:09.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:10 smithi094 ceph-mon[29291]: pgmap v1253: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:10 smithi094 ceph-mon[40484]: pgmap v1253: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:33:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:33:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:33:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:10 smithi151 ceph-mon[32923]: pgmap v1253: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:12.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:12 smithi094 ceph-mon[29291]: pgmap v1254: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:12 smithi094 ceph-mon[40484]: pgmap v1254: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:12 smithi151 ceph-mon[32923]: pgmap v1254: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:14.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:14 smithi094 ceph-mon[40484]: pgmap v1255: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:14 smithi094 ceph-mon[29291]: pgmap v1255: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:14 smithi151 ceph-mon[32923]: pgmap v1255: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:16.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:16 smithi094 ceph-mon[40484]: pgmap v1256: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:16 smithi094 ceph-mon[29291]: pgmap v1256: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:16 smithi151 ceph-mon[32923]: pgmap v1256: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:18.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:18 smithi094 ceph-mon[29291]: pgmap v1257: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:18 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:18 smithi094 ceph-mon[40484]: pgmap v1257: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:18 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:18 smithi151 ceph-mon[32923]: pgmap v1257: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:18 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:18.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:33:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:33:19.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:19.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:19 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:33:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:19.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:19 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:33:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:19.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:19 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:33:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:20 smithi094 ceph-mon[29291]: pgmap v1258: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:33:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:20 smithi094 ceph-mon[40484]: pgmap v1258: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:33:20.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:33:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:33:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:33:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:20 smithi151 ceph-mon[32923]: pgmap v1258: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:33:22.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:22 smithi094 ceph-mon[40484]: pgmap v1259: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:33:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:22 smithi094 ceph-mon[29291]: pgmap v1259: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:33:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:22 smithi151 ceph-mon[32923]: pgmap v1259: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:33:24.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:24 smithi094 ceph-mon[29291]: pgmap v1260: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:24.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:24 smithi094 ceph-mon[40484]: pgmap v1260: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:24 smithi151 ceph-mon[32923]: pgmap v1260: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:26.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:26 smithi094 ceph-mon[29291]: pgmap v1261: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:33:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:26 smithi094 ceph-mon[40484]: pgmap v1261: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:33:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:26 smithi151 ceph-mon[32923]: pgmap v1261: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:33:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:28 smithi094 ceph-mon[29291]: pgmap v1262: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:33:28.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:28 smithi094 ceph-mon[40484]: pgmap v1262: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:33:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:28 smithi151 ceph-mon[32923]: pgmap v1262: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:33:28.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:33:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:33:29.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:30.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:30 smithi094 ceph-mon[40484]: pgmap v1263: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:30 smithi094 ceph-mon[29291]: pgmap v1263: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:30.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:33:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:33:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:33:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:30 smithi151 ceph-mon[32923]: pgmap v1263: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:32 smithi094 ceph-mon[29291]: pgmap v1264: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:32.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:32 smithi094 ceph-mon[40484]: pgmap v1264: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:32 smithi151 ceph-mon[32923]: pgmap v1264: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:34.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:34 smithi094 ceph-mon[29291]: pgmap v1265: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:34 smithi094 ceph-mon[40484]: pgmap v1265: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:34 smithi151 ceph-mon[32923]: pgmap v1265: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:35.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:33:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:33:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:33:36.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:36 smithi094 ceph-mon[29291]: pgmap v1266: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:36 smithi094 ceph-mon[40484]: pgmap v1266: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:36 smithi151 ceph-mon[32923]: pgmap v1266: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:38 smithi094 ceph-mon[29291]: pgmap v1267: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:38 smithi094 ceph-mon[40484]: pgmap v1267: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:38 smithi151 ceph-mon[32923]: pgmap v1267: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:38.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:33:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:33:39.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:40.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:40 smithi094 ceph-mon[29291]: pgmap v1268: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:40 smithi094 ceph-mon[40484]: pgmap v1268: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:33:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:33:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:33:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:40 smithi151 ceph-mon[32923]: pgmap v1268: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:42.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:42 smithi094 ceph-mon[40484]: pgmap v1269: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:42 smithi094 ceph-mon[29291]: pgmap v1269: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:42 smithi151 ceph-mon[32923]: pgmap v1269: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:44 smithi094 ceph-mon[40484]: pgmap v1270: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:44 smithi094 ceph-mon[29291]: pgmap v1270: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:44 smithi151 ceph-mon[32923]: pgmap v1270: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:33:46.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:46 smithi094 ceph-mon[29291]: pgmap v1271: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:46 smithi094 ceph-mon[40484]: pgmap v1271: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:46 smithi151 ceph-mon[32923]: pgmap v1271: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:33:48.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:48 smithi094 ceph-mon[29291]: pgmap v1272: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:48.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:48 smithi094 ceph-mon[40484]: pgmap v1272: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:48 smithi151 ceph-mon[32923]: pgmap v1272: 228 pgs: 228 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:48.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:33:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:33:49.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:49.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:49.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:50 smithi094 ceph-mon[29291]: pgmap v1273: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:33:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:50 smithi094 ceph-mon[40484]: pgmap v1273: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:33:50.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:33:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:33:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:33:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:50 smithi151 ceph-mon[32923]: pgmap v1273: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:33:52.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:52 smithi094 ceph-mon[29291]: pgmap v1274: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:52 smithi094 ceph-mon[40484]: pgmap v1274: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:52.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:52 smithi151 ceph-mon[32923]: pgmap v1274: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:54.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:54 smithi094 ceph-mon[29291]: pgmap v1275: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:54.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:54 smithi094 ceph-mon[40484]: pgmap v1275: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:54 smithi151 ceph-mon[32923]: pgmap v1275: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:56.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:56 smithi094 ceph-mon[29291]: pgmap v1276: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:56.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:56 smithi094 ceph-mon[40484]: pgmap v1276: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:56.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:56 smithi151 ceph-mon[32923]: pgmap v1276: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:58.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:58 smithi094 ceph-mon[29291]: pgmap v1277: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:58 smithi094 ceph-mon[40484]: pgmap v1277: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:58 smithi151 ceph-mon[32923]: pgmap v1277: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:33:58.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:33:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:33:59.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:33:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:59.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:33:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:59.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:33:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:00 smithi094 ceph-mon[29291]: pgmap v1278: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:34:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:00 smithi094 ceph-mon[40484]: pgmap v1278: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:34:00.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:34:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:34:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:34:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:00 smithi151 ceph-mon[32923]: pgmap v1278: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:34:02.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:02 smithi094 ceph-mon[29291]: pgmap v1279: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:02 smithi094 ceph-mon[40484]: pgmap v1279: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:02 smithi151 ceph-mon[32923]: pgmap v1279: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:04 smithi094 ceph-mon[29291]: pgmap v1280: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:04 smithi094 ceph-mon[40484]: pgmap v1280: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:04 smithi151 ceph-mon[32923]: pgmap v1280: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:05.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:06.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:06 smithi094 ceph-mon[29291]: pgmap v1281: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:06 smithi094 ceph-mon[40484]: pgmap v1281: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:06 smithi151 ceph-mon[32923]: pgmap v1281: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:08 smithi094 ceph-mon[29291]: pgmap v1282: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:08 smithi094 ceph-mon[40484]: pgmap v1282: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:08 smithi151 ceph-mon[32923]: pgmap v1282: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:08.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:34:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:34:09.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:09.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:09.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:10.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:10 smithi094 ceph-mon[29291]: pgmap v1283: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:10 smithi094 ceph-mon[40484]: pgmap v1283: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:34:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:34:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:34:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:10 smithi151 ceph-mon[32923]: pgmap v1283: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:12 smithi151 ceph-mon[32923]: pgmap v1284: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:13.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:12 smithi094 ceph-mon[29291]: pgmap v1284: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:13.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:12 smithi094 ceph-mon[40484]: pgmap v1284: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:14 smithi151 ceph-mon[32923]: pgmap v1285: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:15.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:14 smithi094 ceph-mon[29291]: pgmap v1285: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:15.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:14 smithi094 ceph-mon[40484]: pgmap v1285: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:17.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:16 smithi094 ceph-mon[29291]: pgmap v1286: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:17.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:16 smithi094 ceph-mon[40484]: pgmap v1286: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:17.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:16 smithi151 ceph-mon[32923]: pgmap v1286: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:18 smithi151 ceph-mon[32923]: pgmap v1287: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:18 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:18.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:34:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:34:19.011 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:18 smithi094 ceph-mon[40484]: pgmap v1287: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:19.011 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:18 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:19.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:18 smithi094 ceph-mon[29291]: pgmap v1287: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:19.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:18 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:20.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:19 smithi094 ceph-mon[40484]: pgmap v1288: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:20.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:20.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:20.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:19 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:20.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:19 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:34:20.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:19 smithi094 ceph-mon[29291]: pgmap v1288: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:20.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:20.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:20.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:19 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:20.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:19 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:34:20.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:19 smithi151 ceph-mon[32923]: pgmap v1288: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:20.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:20.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:20.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:19 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:20.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:19 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:34:20.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:20.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:34:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:34:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:34:21.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:22.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:21 smithi094 ceph-mon[29291]: pgmap v1289: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:22.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:21 smithi094 ceph-mon[40484]: pgmap v1289: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:22.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:21 smithi151 ceph-mon[32923]: pgmap v1289: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:24.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:23 smithi151 ceph-mon[32923]: pgmap v1290: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:24.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:23 smithi094 ceph-mon[29291]: pgmap v1290: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:24.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:23 smithi094 ceph-mon[40484]: pgmap v1290: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:26.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:25 smithi094 ceph-mon[29291]: pgmap v1291: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:26.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:25 smithi094 ceph-mon[40484]: pgmap v1291: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:26.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:25 smithi151 ceph-mon[32923]: pgmap v1291: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:28.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:27 smithi094 ceph-mon[40484]: pgmap v1292: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:28.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:27 smithi094 ceph-mon[29291]: pgmap v1292: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:28.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:27 smithi151 ceph-mon[32923]: pgmap v1292: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:28.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:34:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:34:29.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:28 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:29.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:28 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:29.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:28 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:30.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:29 smithi094 ceph-mon[29291]: pgmap v1293: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:30.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:29 smithi094 ceph-mon[40484]: pgmap v1293: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:30.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:29 smithi151 ceph-mon[32923]: pgmap v1293: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:30.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:34:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:34:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:34:32.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:31 smithi094 ceph-mon[29291]: pgmap v1294: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:32.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:31 smithi094 ceph-mon[40484]: pgmap v1294: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:32.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:31 smithi151 ceph-mon[32923]: pgmap v1294: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:34.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:33 smithi094 ceph-mon[40484]: pgmap v1295: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:34.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:33 smithi094 ceph-mon[29291]: pgmap v1295: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:34.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:33 smithi151 ceph-mon[32923]: pgmap v1295: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:35.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:35.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:35.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:36.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:35 smithi094 ceph-mon[29291]: pgmap v1296: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:36.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:35 smithi094 ceph-mon[40484]: pgmap v1296: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:36.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:35 smithi151 ceph-mon[32923]: pgmap v1296: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:38.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:37 smithi094 ceph-mon[29291]: pgmap v1297: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:38.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:37 smithi094 ceph-mon[40484]: pgmap v1297: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:38.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:37 smithi151 ceph-mon[32923]: pgmap v1297: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:38.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:34:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:34:39.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:38 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:39.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:38 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:39.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:38 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:40.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:39 smithi094 ceph-mon[29291]: pgmap v1298: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:40.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:39 smithi094 ceph-mon[40484]: pgmap v1298: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:40.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:39 smithi151 ceph-mon[32923]: pgmap v1298: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:40.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:34:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:34:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:34:42.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:41 smithi094 ceph-mon[29291]: pgmap v1299: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:42.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:41 smithi094 ceph-mon[40484]: pgmap v1299: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:42.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:42 smithi151 ceph-mon[32923]: pgmap v1299: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:44.266 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:44 smithi151 ceph-mon[32923]: pgmap v1300: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:44.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:44 smithi094 ceph-mon[29291]: pgmap v1300: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:44.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:44 smithi094 ceph-mon[40484]: pgmap v1300: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:46.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:46 smithi094 ceph-mon[29291]: pgmap v1301: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:46.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:46 smithi094 ceph-mon[40484]: pgmap v1301: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:46.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:46 smithi151 ceph-mon[32923]: pgmap v1301: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:48.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:48 smithi094 ceph-mon[29291]: pgmap v1302: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:48.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:48 smithi094 ceph-mon[40484]: pgmap v1302: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:48.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:48 smithi151 ceph-mon[32923]: pgmap v1302: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:48.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:34:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:34:49.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:49.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:49.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:50.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:34:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:34:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:34:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:50 smithi094 ceph-mon[40484]: pgmap v1303: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:50 smithi094 ceph-mon[29291]: pgmap v1303: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:50 smithi151 ceph-mon[32923]: pgmap v1303: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:52.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:52 smithi094 ceph-mon[29291]: pgmap v1304: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:52.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:52 smithi094 ceph-mon[40484]: pgmap v1304: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:52 smithi151 ceph-mon[32923]: pgmap v1304: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:54.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:54 smithi094 ceph-mon[29291]: pgmap v1305: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:54.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:54 smithi094 ceph-mon[40484]: pgmap v1305: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:54 smithi151 ceph-mon[32923]: pgmap v1305: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:56.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:56 smithi094 ceph-mon[29291]: pgmap v1306: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:56 smithi094 ceph-mon[40484]: pgmap v1306: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:56 smithi151 ceph-mon[32923]: pgmap v1306: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:34:58.682 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:58 smithi151 ceph-mon[32923]: pgmap v1307: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:58.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:58 smithi094 ceph-mon[29291]: pgmap v1307: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:58.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:58 smithi094 ceph-mon[40484]: pgmap v1307: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:58.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:34:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:34:59.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:34:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:59.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:34:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:59.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:34:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:00.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:00 smithi094 ceph-mon[29291]: pgmap v1308: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:00.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:35:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:35:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:35:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:00 smithi094 ceph-mon[40484]: pgmap v1308: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:00 smithi151 ceph-mon[32923]: pgmap v1308: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:02.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:02 smithi094 ceph-mon[29291]: pgmap v1309: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:35:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:02 smithi094 ceph-mon[40484]: pgmap v1309: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:35:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:02 smithi151 ceph-mon[32923]: pgmap v1309: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:35:04.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:04 smithi094 ceph-mon[29291]: pgmap v1310: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:04 smithi094 ceph-mon[40484]: pgmap v1310: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:04 smithi151 ceph-mon[32923]: pgmap v1310: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:05.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:06.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:06 smithi094 ceph-mon[29291]: pgmap v1311: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:06.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:06 smithi094 ceph-mon[40484]: pgmap v1311: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:06 smithi151 ceph-mon[32923]: pgmap v1311: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:08.693 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:08 smithi151 ceph-mon[32923]: pgmap v1312: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:08.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:08 smithi094 ceph-mon[40484]: pgmap v1312: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:08.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:08 smithi094 ceph-mon[29291]: pgmap v1312: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:08.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:35:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:35:09.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:09.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:09.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:10 smithi094 ceph-mon[29291]: pgmap v1313: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:10 smithi094 ceph-mon[40484]: pgmap v1313: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:10.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:35:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:35:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:35:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:10 smithi151 ceph-mon[32923]: pgmap v1313: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:12.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:12 smithi094 ceph-mon[29291]: pgmap v1314: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:12.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:12 smithi094 ceph-mon[40484]: pgmap v1314: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:12 smithi151 ceph-mon[32923]: pgmap v1314: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:14.397 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:35:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=cleanup t=2024-03-12T03:35:14.016129912Z level=info msg="Completed cleanup jobs" duration=3.197896ms 2024-03-12T03:35:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:14 smithi094 ceph-mon[29291]: pgmap v1315: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:14 smithi094 ceph-mon[40484]: pgmap v1315: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:14 smithi151 ceph-mon[32923]: pgmap v1315: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:16.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:16 smithi094 ceph-mon[40484]: pgmap v1316: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:16.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:16 smithi094 ceph-mon[29291]: pgmap v1316: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:16.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:16 smithi151 ceph-mon[32923]: pgmap v1316: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:18.704 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:18 smithi151 ceph-mon[32923]: pgmap v1317: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:18.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:18 smithi094 ceph-mon[40484]: pgmap v1317: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:18.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:18 smithi094 ceph-mon[29291]: pgmap v1317: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:18.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:35:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:35:19.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:19 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:19.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:19 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:19.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:19 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:20.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:35:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:35:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:35:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:20 smithi094 ceph-mon[29291]: pgmap v1318: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:35:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:35:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:35:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:20 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:35:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:20 smithi094 ceph-mon[40484]: pgmap v1318: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:35:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:35:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:35:20.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:20 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:35:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:20 smithi151 ceph-mon[32923]: pgmap v1318: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:35:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:35:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:35:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:20 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:35:22.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:22 smithi094 ceph-mon[29291]: pgmap v1319: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:22.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:22 smithi094 ceph-mon[40484]: pgmap v1319: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:22 smithi151 ceph-mon[32923]: pgmap v1319: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:24.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:24 smithi094 ceph-mon[29291]: pgmap v1320: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:24.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:24 smithi094 ceph-mon[40484]: pgmap v1320: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:24 smithi151 ceph-mon[32923]: pgmap v1320: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:26.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:26 smithi094 ceph-mon[29291]: pgmap v1321: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:26.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:26 smithi094 ceph-mon[40484]: pgmap v1321: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:26 smithi151 ceph-mon[32923]: pgmap v1321: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:28.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:28 smithi094 ceph-mon[29291]: pgmap v1322: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:28 smithi094 ceph-mon[40484]: pgmap v1322: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:28.710 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:28 smithi151 ceph-mon[32923]: pgmap v1322: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:28.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:35:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:35:29.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:29 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:29.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:29 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:29.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:29 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:30.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:30 smithi094 ceph-mon[29291]: pgmap v1323: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:30 smithi094 ceph-mon[40484]: pgmap v1323: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:30.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:35:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:35:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:35:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:30 smithi151 ceph-mon[32923]: pgmap v1323: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:32.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:32 smithi094 ceph-mon[29291]: pgmap v1324: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:35:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:32 smithi094 ceph-mon[40484]: pgmap v1324: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:35:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:32 smithi151 ceph-mon[32923]: pgmap v1324: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:35:34.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:34 smithi094 ceph-mon[29291]: pgmap v1325: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:34 smithi094 ceph-mon[40484]: pgmap v1325: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:34 smithi151 ceph-mon[32923]: pgmap v1325: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:35.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:35.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:36.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:36 smithi094 ceph-mon[29291]: pgmap v1326: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:35:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:36 smithi094 ceph-mon[40484]: pgmap v1326: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:35:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:36 smithi151 ceph-mon[32923]: pgmap v1326: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:35:38.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:38 smithi094 ceph-mon[29291]: pgmap v1327: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:38 smithi094 ceph-mon[40484]: pgmap v1327: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:38.711 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:38 smithi151 ceph-mon[32923]: pgmap v1327: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:38.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:35:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:35:39.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:39 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:39.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:39 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:39.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:39 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:40 smithi094 ceph-mon[29291]: pgmap v1328: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:40.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:40 smithi094 ceph-mon[40484]: pgmap v1328: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:40.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:35:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:35:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:35:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:40 smithi151 ceph-mon[32923]: pgmap v1328: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:42.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:42 smithi094 ceph-mon[29291]: pgmap v1329: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:35:42.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:42 smithi094 ceph-mon[40484]: pgmap v1329: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:35:42.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:42 smithi151 ceph-mon[32923]: pgmap v1329: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:35:44.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:44 smithi094 ceph-mon[29291]: pgmap v1330: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:44 smithi094 ceph-mon[40484]: pgmap v1330: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:44 smithi151 ceph-mon[32923]: pgmap v1330: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:46.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:46 smithi094 ceph-mon[29291]: pgmap v1331: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:46 smithi094 ceph-mon[40484]: pgmap v1331: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:46 smithi151 ceph-mon[32923]: pgmap v1331: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:48.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:48 smithi094 ceph-mon[29291]: pgmap v1332: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:48 smithi094 ceph-mon[40484]: pgmap v1332: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:48 smithi151 ceph-mon[32923]: pgmap v1332: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:49.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:35:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:35:49.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:49 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:49.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:49 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:49.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:49 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:50 smithi094 ceph-mon[29291]: pgmap v1333: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:50.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:35:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:35:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:35:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:50 smithi094 ceph-mon[40484]: pgmap v1333: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:50 smithi151 ceph-mon[32923]: pgmap v1333: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:52.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:52 smithi094 ceph-mon[29291]: pgmap v1334: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:52 smithi094 ceph-mon[40484]: pgmap v1334: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:52.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:52 smithi151 ceph-mon[32923]: pgmap v1334: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:54.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:54 smithi094 ceph-mon[40484]: pgmap v1335: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:54.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:54 smithi094 ceph-mon[29291]: pgmap v1335: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:54 smithi151 ceph-mon[32923]: pgmap v1335: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:56.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:56 smithi094 ceph-mon[40484]: pgmap v1336: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:56.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:56 smithi094 ceph-mon[29291]: pgmap v1336: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:56.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:56 smithi151 ceph-mon[32923]: pgmap v1336: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:58.732 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:35:58 smithi151 ceph-mon[32923]: pgmap v1337: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:58.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:35:58 smithi094 ceph-mon[29291]: pgmap v1337: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:35:58 smithi094 ceph-mon[40484]: pgmap v1337: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:59.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:35:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:36:00.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:36:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:36:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:36:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:00 smithi094 ceph-mon[29291]: pgmap v1338: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:00 smithi094 ceph-mon[40484]: pgmap v1338: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:00 smithi151 ceph-mon[32923]: pgmap v1338: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:02.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:02 smithi094 ceph-mon[29291]: pgmap v1339: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:36:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:02 smithi094 ceph-mon[40484]: pgmap v1339: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:36:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:02 smithi151 ceph-mon[32923]: pgmap v1339: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:36:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:04 smithi094 ceph-mon[29291]: pgmap v1340: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:04 smithi094 ceph-mon[40484]: pgmap v1340: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:04 smithi151 ceph-mon[32923]: pgmap v1340: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:05.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:06 smithi094 ceph-mon[29291]: pgmap v1341: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:36:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:06 smithi094 ceph-mon[40484]: pgmap v1341: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:36:06.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:06 smithi151 ceph-mon[32923]: pgmap v1341: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:36:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:08 smithi094 ceph-mon[29291]: pgmap v1342: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:08 smithi094 ceph-mon[40484]: pgmap v1342: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:08 smithi151 ceph-mon[32923]: pgmap v1342: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:08.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:36:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:36:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:36:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:36:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:36:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:10 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:10 smithi094 ceph-mon[29291]: pgmap v1343: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:10 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:10 smithi094 ceph-mon[40484]: pgmap v1343: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:10 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:10 smithi151 ceph-mon[32923]: pgmap v1343: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:12.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:12 smithi094 ceph-mon[29291]: pgmap v1344: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:36:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:12 smithi094 ceph-mon[40484]: pgmap v1344: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:36:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:12 smithi151 ceph-mon[32923]: pgmap v1344: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:36:14.767 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:14 smithi151 ceph-mon[32923]: pgmap v1345: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:14.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:14 smithi094 ceph-mon[29291]: pgmap v1345: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:14.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:14 smithi094 ceph-mon[40484]: pgmap v1345: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:16.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:16 smithi094 ceph-mon[29291]: pgmap v1346: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:36:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:16 smithi094 ceph-mon[40484]: pgmap v1346: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:36:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:16 smithi151 ceph-mon[32923]: pgmap v1346: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:36:18.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:18 smithi094 ceph-mon[29291]: pgmap v1347: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:18.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:18 smithi094 ceph-mon[40484]: pgmap v1347: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:18 smithi151 ceph-mon[32923]: pgmap v1347: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:18.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:36:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:36:20.761 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:20.761 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:20 smithi094 ceph-mon[40484]: pgmap v1348: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:20.762 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:20.762 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:36:20.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:20.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:20 smithi094 ceph-mon[29291]: pgmap v1348: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:20.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:20.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:36:20.762 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:36:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:36:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:36:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:20 smithi151 ceph-mon[32923]: pgmap v1348: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:36:21.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:21 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:21.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:21 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:21 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:36:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:21 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:21 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:21 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:36:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:21 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:21 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:21 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:36:22.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:22 smithi094 ceph-mon[40484]: pgmap v1349: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:22 smithi094 ceph-mon[29291]: pgmap v1349: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:22 smithi151 ceph-mon[32923]: pgmap v1349: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:24.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:24 smithi094 ceph-mon[29291]: pgmap v1350: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:24.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:24 smithi094 ceph-mon[40484]: pgmap v1350: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:24 smithi151 ceph-mon[32923]: pgmap v1350: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:26.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:26 smithi094 ceph-mon[29291]: pgmap v1351: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:26 smithi094 ceph-mon[40484]: pgmap v1351: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:26 smithi151 ceph-mon[32923]: pgmap v1351: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:28 smithi094 ceph-mon[29291]: pgmap v1352: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:28 smithi094 ceph-mon[40484]: pgmap v1352: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:28 smithi151 ceph-mon[32923]: pgmap v1352: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:28.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:36:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:36:30.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:30 smithi094 ceph-mon[29291]: pgmap v1353: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:30.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:36:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:36:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:36:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:30 smithi094 ceph-mon[40484]: pgmap v1353: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:30 smithi151 ceph-mon[32923]: pgmap v1353: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:32 smithi094 ceph-mon[29291]: pgmap v1354: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:32.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:32 smithi094 ceph-mon[40484]: pgmap v1354: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:32 smithi151 ceph-mon[32923]: pgmap v1354: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:34.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:34 smithi094 ceph-mon[29291]: pgmap v1355: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:34 smithi094 ceph-mon[40484]: pgmap v1355: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:34 smithi151 ceph-mon[32923]: pgmap v1355: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:35.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:36.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:36 smithi094 ceph-mon[29291]: pgmap v1356: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:36 smithi094 ceph-mon[40484]: pgmap v1356: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:36 smithi151 ceph-mon[32923]: pgmap v1356: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:38.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:38 smithi094 ceph-mon[40484]: pgmap v1357: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:38.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:38 smithi094 ceph-mon[29291]: pgmap v1357: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:38 smithi151 ceph-mon[32923]: pgmap v1357: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:38.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:36:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:36:40.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:36:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:36:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:36:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:40 smithi094 ceph-mon[40484]: pgmap v1358: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:40 smithi094 ceph-mon[29291]: pgmap v1358: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:40 smithi151 ceph-mon[32923]: pgmap v1358: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:42.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:42 smithi094 ceph-mon[29291]: pgmap v1359: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:42 smithi094 ceph-mon[40484]: pgmap v1359: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:42 smithi151 ceph-mon[32923]: pgmap v1359: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:44.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:44 smithi094 ceph-mon[29291]: pgmap v1360: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:44 smithi094 ceph-mon[40484]: pgmap v1360: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:44 smithi151 ceph-mon[32923]: pgmap v1360: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:46.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:46 smithi094 ceph-mon[29291]: pgmap v1361: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:46 smithi094 ceph-mon[40484]: pgmap v1361: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:46 smithi151 ceph-mon[32923]: pgmap v1361: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:48 smithi094 ceph-mon[29291]: pgmap v1362: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:48.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:48 smithi094 ceph-mon[40484]: pgmap v1362: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:48 smithi151 ceph-mon[32923]: pgmap v1362: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:48.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:36:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:36:50.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:36:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:36:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:36:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:50 smithi094 ceph-mon[29291]: pgmap v1363: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:50 smithi094 ceph-mon[40484]: pgmap v1363: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:50.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:50 smithi151 ceph-mon[32923]: pgmap v1363: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:50.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:52.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:52 smithi094 ceph-mon[29291]: pgmap v1364: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:52 smithi094 ceph-mon[40484]: pgmap v1364: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:52.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:52 smithi151 ceph-mon[32923]: pgmap v1364: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:54.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:54 smithi094 ceph-mon[29291]: pgmap v1365: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:54.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:54 smithi094 ceph-mon[40484]: pgmap v1365: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:54 smithi151 ceph-mon[32923]: pgmap v1365: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:56.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:56 smithi094 ceph-mon[29291]: pgmap v1366: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:56.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:56 smithi094 ceph-mon[40484]: pgmap v1366: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:56.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:56 smithi151 ceph-mon[32923]: pgmap v1366: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:58.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:36:58 smithi094 ceph-mon[29291]: pgmap v1367: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:36:58 smithi094 ceph-mon[40484]: pgmap v1367: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:36:58 smithi151 ceph-mon[32923]: pgmap v1367: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:36:58.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:36:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:37:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:00 smithi094 ceph-mon[29291]: pgmap v1368: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:00.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:37:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:37:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:37:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:00 smithi094 ceph-mon[40484]: pgmap v1368: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:00 smithi151 ceph-mon[32923]: pgmap v1368: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:02.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:02 smithi094 ceph-mon[29291]: pgmap v1369: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:02 smithi094 ceph-mon[40484]: pgmap v1369: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:02 smithi151 ceph-mon[32923]: pgmap v1369: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:04.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:04 smithi094 ceph-mon[40484]: pgmap v1370: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:04.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:04 smithi094 ceph-mon[29291]: pgmap v1370: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:04 smithi151 ceph-mon[32923]: pgmap v1370: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:05.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:06 smithi094 ceph-mon[29291]: pgmap v1371: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:06 smithi094 ceph-mon[40484]: pgmap v1371: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:06 smithi151 ceph-mon[32923]: pgmap v1371: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:08 smithi094 ceph-mon[29291]: pgmap v1372: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:08 smithi094 ceph-mon[40484]: pgmap v1372: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:08 smithi151 ceph-mon[32923]: pgmap v1372: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:08.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:37:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:37:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:10 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:10 smithi094 ceph-mon[40484]: pgmap v1373: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:10 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:10 smithi094 ceph-mon[29291]: pgmap v1373: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:37:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:37:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:37:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:10 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:10 smithi151 ceph-mon[32923]: pgmap v1373: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:12.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:12 smithi094 ceph-mon[29291]: pgmap v1374: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:12.963 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:12 smithi094 ceph-mon[40484]: pgmap v1374: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:12 smithi151 ceph-mon[32923]: pgmap v1374: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:14.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:14 smithi094 ceph-mon[29291]: pgmap v1375: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:14 smithi094 ceph-mon[40484]: pgmap v1375: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:14 smithi151 ceph-mon[32923]: pgmap v1375: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:16.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:16 smithi094 ceph-mon[29291]: pgmap v1376: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:16 smithi094 ceph-mon[40484]: pgmap v1376: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:16 smithi151 ceph-mon[32923]: pgmap v1376: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:18.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:18 smithi094 ceph-mon[29291]: pgmap v1377: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:18.968 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:18 smithi094 ceph-mon[40484]: pgmap v1377: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:18.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:18 smithi151 ceph-mon[32923]: pgmap v1377: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:18.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:37:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:37:20.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:20 smithi094 ceph-mon[40484]: pgmap v1378: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:20 smithi094 ceph-mon[29291]: pgmap v1378: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:20.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:37:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:37:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:37:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:20 smithi151 ceph-mon[32923]: pgmap v1378: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:21.927 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:21 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:37:21.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:21 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:37:21.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:21 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:37:22.868 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:22 smithi094 ceph-mon[40484]: pgmap v1379: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:22.868 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:22 smithi094 ceph-mon[29291]: pgmap v1379: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:22 smithi151 ceph-mon[32923]: pgmap v1379: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:24.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:24 smithi094 ceph-mon[40484]: pgmap v1380: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:24.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:24 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:24.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:24 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:24.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:24 smithi094 ceph-mon[29291]: pgmap v1380: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:24.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:24 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:24.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:24 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:24 smithi151 ceph-mon[32923]: pgmap v1380: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:24 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:24 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:26.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:26 smithi094 ceph-mon[29291]: pgmap v1381: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:26.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:26 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:26.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:26 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:26.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:26 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:37:26.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:26 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:37:26.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:26 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:26 smithi094 ceph-mon[40484]: pgmap v1381: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:26 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:26 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:26 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:37:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:26 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:37:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:26 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:26 smithi151 ceph-mon[32923]: pgmap v1381: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:26 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:26 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:26 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:37:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:26 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:37:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:26 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:37:28.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:28 smithi094 ceph-mon[29291]: pgmap v1382: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:28 smithi094 ceph-mon[40484]: pgmap v1382: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:28 smithi151 ceph-mon[32923]: pgmap v1382: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:29.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:37:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:37:30.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:30.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:30 smithi094 ceph-mon[29291]: pgmap v1383: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:30.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:37:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:37:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:37:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:30.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:30 smithi094 ceph-mon[40484]: pgmap v1383: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:30 smithi151 ceph-mon[32923]: pgmap v1383: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:32.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:32 smithi094 ceph-mon[29291]: pgmap v1384: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:32 smithi094 ceph-mon[40484]: pgmap v1384: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:32 smithi151 ceph-mon[32923]: pgmap v1384: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:34.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:34 smithi094 ceph-mon[29291]: pgmap v1385: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:34 smithi094 ceph-mon[40484]: pgmap v1385: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:34 smithi151 ceph-mon[32923]: pgmap v1385: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:35.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:35.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:36.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:36 smithi094 ceph-mon[29291]: pgmap v1386: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:36 smithi094 ceph-mon[40484]: pgmap v1386: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:36 smithi151 ceph-mon[32923]: pgmap v1386: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:38.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:38 smithi094 ceph-mon[40484]: pgmap v1387: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:38.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:38 smithi094 ceph-mon[29291]: pgmap v1387: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:38.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:38 smithi151 ceph-mon[32923]: pgmap v1387: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:39.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:37:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:37:40.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:40.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:40 smithi094 ceph-mon[40484]: pgmap v1388: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:40.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:40 smithi094 ceph-mon[29291]: pgmap v1388: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:40.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:37:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:37:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:37:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:40.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:40 smithi151 ceph-mon[32923]: pgmap v1388: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:42.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:42 smithi094 ceph-mon[29291]: pgmap v1389: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:37:42.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:42 smithi094 ceph-mon[40484]: pgmap v1389: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:37:42.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:42 smithi151 ceph-mon[32923]: pgmap v1389: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:37:44.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:44 smithi094 ceph-mon[29291]: pgmap v1390: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:44 smithi094 ceph-mon[40484]: pgmap v1390: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:44 smithi151 ceph-mon[32923]: pgmap v1390: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:46.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:46 smithi094 ceph-mon[29291]: pgmap v1391: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:37:46.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:46 smithi094 ceph-mon[40484]: pgmap v1391: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:37:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:46 smithi151 ceph-mon[32923]: pgmap v1391: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:37:48.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:48 smithi094 ceph-mon[29291]: pgmap v1392: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:48.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:48 smithi094 ceph-mon[40484]: pgmap v1392: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:48 smithi151 ceph-mon[32923]: pgmap v1392: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:49.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:37:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:37:50.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:37:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:37:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:37:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:50 smithi094 ceph-mon[29291]: pgmap v1393: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:50.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:50 smithi094 ceph-mon[40484]: pgmap v1393: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:50.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:50 smithi151 ceph-mon[32923]: pgmap v1393: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:52.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:52 smithi094 ceph-mon[29291]: pgmap v1394: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:37:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:52 smithi094 ceph-mon[40484]: pgmap v1394: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:37:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:52 smithi151 ceph-mon[32923]: pgmap v1394: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:37:54.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:54 smithi094 ceph-mon[40484]: pgmap v1395: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:54.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:54 smithi094 ceph-mon[29291]: pgmap v1395: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:54 smithi151 ceph-mon[32923]: pgmap v1395: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:56.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:56 smithi094 ceph-mon[29291]: pgmap v1396: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:56 smithi094 ceph-mon[40484]: pgmap v1396: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:56 smithi151 ceph-mon[32923]: pgmap v1396: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:37:58 smithi094 ceph-mon[29291]: pgmap v1397: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:37:58 smithi094 ceph-mon[40484]: pgmap v1397: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:37:58 smithi151 ceph-mon[32923]: pgmap v1397: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:59.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:37:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:38:00.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:00 smithi094 ceph-mon[40484]: pgmap v1398: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:00.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:00.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:00 smithi094 ceph-mon[29291]: pgmap v1398: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:00.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:38:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:38:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:38:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:00.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:00 smithi151 ceph-mon[32923]: pgmap v1398: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:02.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:02 smithi094 ceph-mon[29291]: pgmap v1399: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:02 smithi094 ceph-mon[40484]: pgmap v1399: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:02 smithi151 ceph-mon[32923]: pgmap v1399: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:04.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:04 smithi094 ceph-mon[29291]: pgmap v1400: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:04 smithi094 ceph-mon[40484]: pgmap v1400: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:04 smithi151 ceph-mon[32923]: pgmap v1400: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:05.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:06.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:06 smithi094 ceph-mon[29291]: pgmap v1401: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:06.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:06 smithi094 ceph-mon[40484]: pgmap v1401: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:06 smithi151 ceph-mon[32923]: pgmap v1401: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:08.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:08 smithi094 ceph-mon[29291]: pgmap v1402: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:08 smithi094 ceph-mon[40484]: pgmap v1402: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:08 smithi151 ceph-mon[32923]: pgmap v1402: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:09.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:38:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:38:10.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:10 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:10 smithi094 ceph-mon[29291]: pgmap v1403: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:10.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:38:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:38:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:38:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:10 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:10 smithi094 ceph-mon[40484]: pgmap v1403: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:10 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:10 smithi151 ceph-mon[32923]: pgmap v1403: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:12.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:12 smithi094 ceph-mon[29291]: pgmap v1404: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:12.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:12 smithi094 ceph-mon[40484]: pgmap v1404: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:12 smithi151 ceph-mon[32923]: pgmap v1404: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:14.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:14 smithi094 ceph-mon[29291]: pgmap v1405: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:14 smithi094 ceph-mon[40484]: pgmap v1405: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:14 smithi151 ceph-mon[32923]: pgmap v1405: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:16.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:16 smithi094 ceph-mon[29291]: pgmap v1406: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:16.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:16 smithi094 ceph-mon[40484]: pgmap v1406: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:16.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:16 smithi151 ceph-mon[32923]: pgmap v1406: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:18.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:18 smithi094 ceph-mon[29291]: pgmap v1407: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:18 smithi094 ceph-mon[40484]: pgmap v1407: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:18 smithi151 ceph-mon[32923]: pgmap v1407: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:19.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:38:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:38:20.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:38:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:38:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:38:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:20 smithi094 ceph-mon[29291]: pgmap v1408: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:20 smithi094 ceph-mon[40484]: pgmap v1408: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:20 smithi151 ceph-mon[32923]: pgmap v1408: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:22.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:22 smithi094 ceph-mon[29291]: pgmap v1409: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:22.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:22 smithi094 ceph-mon[40484]: pgmap v1409: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:22 smithi151 ceph-mon[32923]: pgmap v1409: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:24.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:24 smithi094 ceph-mon[29291]: pgmap v1410: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:24.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:24 smithi094 ceph-mon[40484]: pgmap v1410: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:24 smithi151 ceph-mon[32923]: pgmap v1410: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:26.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:26 smithi094 ceph-mon[29291]: pgmap v1411: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:26.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:26 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:38:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:26 smithi094 ceph-mon[40484]: pgmap v1411: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:26.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:26 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:38:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:26 smithi151 ceph-mon[32923]: pgmap v1411: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:26 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:38:27.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:27 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:38:27.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:27 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:38:27.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:27 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:38:27.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:27 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:38:27.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:27 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:38:27.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:27 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:38:27.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:27 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:38:27.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:27 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:38:27.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:27 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:38:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:28 smithi094 ceph-mon[29291]: pgmap v1412: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:28 smithi094 ceph-mon[40484]: pgmap v1412: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:28 smithi151 ceph-mon[32923]: pgmap v1412: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:28.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:38:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:38:30.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:30 smithi094 ceph-mon[40484]: pgmap v1413: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:30 smithi094 ceph-mon[29291]: pgmap v1413: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:30.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:38:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:38:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:38:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:30 smithi151 ceph-mon[32923]: pgmap v1413: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:32 smithi094 ceph-mon[29291]: pgmap v1414: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:32.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:32 smithi094 ceph-mon[40484]: pgmap v1414: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:32 smithi151 ceph-mon[32923]: pgmap v1414: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:34.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:34 smithi094 ceph-mon[29291]: pgmap v1415: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:34 smithi094 ceph-mon[40484]: pgmap v1415: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:34 smithi151 ceph-mon[32923]: pgmap v1415: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:35.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:36.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:36 smithi094 ceph-mon[40484]: pgmap v1416: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:36.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:36 smithi094 ceph-mon[29291]: pgmap v1416: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:36 smithi151 ceph-mon[32923]: pgmap v1416: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:38 smithi094 ceph-mon[29291]: pgmap v1417: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:38 smithi094 ceph-mon[40484]: pgmap v1417: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:38 smithi151 ceph-mon[32923]: pgmap v1417: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:38.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:38:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:38:40.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:40 smithi094 ceph-mon[29291]: pgmap v1418: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:38:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:38:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:38:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:40 smithi094 ceph-mon[40484]: pgmap v1418: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:40 smithi151 ceph-mon[32923]: pgmap v1418: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:42 smithi094 ceph-mon[29291]: pgmap v1419: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:42 smithi094 ceph-mon[40484]: pgmap v1419: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:42 smithi151 ceph-mon[32923]: pgmap v1419: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:44.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:44 smithi094 ceph-mon[29291]: pgmap v1420: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:44 smithi094 ceph-mon[40484]: pgmap v1420: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:44 smithi151 ceph-mon[32923]: pgmap v1420: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:46.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:46 smithi094 ceph-mon[29291]: pgmap v1421: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:46 smithi094 ceph-mon[40484]: pgmap v1421: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:46 smithi151 ceph-mon[32923]: pgmap v1421: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:48.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:48 smithi094 ceph-mon[29291]: pgmap v1422: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:48.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:48 smithi094 ceph-mon[40484]: pgmap v1422: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:48 smithi151 ceph-mon[32923]: pgmap v1422: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:48.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:38:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:38:50.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:50 smithi094 ceph-mon[40484]: pgmap v1423: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:50 smithi094 ceph-mon[29291]: pgmap v1423: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:50.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:38:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:38:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:38:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:50 smithi151 ceph-mon[32923]: pgmap v1423: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:52.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:52 smithi094 ceph-mon[29291]: pgmap v1424: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:52 smithi094 ceph-mon[40484]: pgmap v1424: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:52.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:52 smithi151 ceph-mon[32923]: pgmap v1424: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:54.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:54 smithi094 ceph-mon[29291]: pgmap v1425: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:54.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:54 smithi094 ceph-mon[40484]: pgmap v1425: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:54 smithi151 ceph-mon[32923]: pgmap v1425: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:56.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:56 smithi094 ceph-mon[29291]: pgmap v1426: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:56.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:56 smithi094 ceph-mon[40484]: pgmap v1426: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:56.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:56 smithi151 ceph-mon[32923]: pgmap v1426: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:38:58.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:38:58 smithi094 ceph-mon[29291]: pgmap v1427: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:38:58 smithi094 ceph-mon[40484]: pgmap v1427: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:38:58 smithi151 ceph-mon[32923]: pgmap v1427: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:58.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:38:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:39:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:00 smithi094 ceph-mon[29291]: pgmap v1428: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:00.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:39:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:39:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:39:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:00 smithi094 ceph-mon[40484]: pgmap v1428: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:00 smithi151 ceph-mon[32923]: pgmap v1428: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:02.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:02 smithi094 ceph-mon[29291]: pgmap v1429: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:02 smithi094 ceph-mon[40484]: pgmap v1429: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:02 smithi151 ceph-mon[32923]: pgmap v1429: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:04 smithi094 ceph-mon[29291]: pgmap v1430: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:04 smithi094 ceph-mon[40484]: pgmap v1430: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:04 smithi151 ceph-mon[32923]: pgmap v1430: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:05.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:06 smithi151 ceph-mon[32923]: pgmap v1431: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:07.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:06 smithi094 ceph-mon[29291]: pgmap v1431: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:07.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:06 smithi094 ceph-mon[40484]: pgmap v1431: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:08 smithi151 ceph-mon[32923]: pgmap v1432: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:08.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:39:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:39:09.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:08 smithi094 ceph-mon[29291]: pgmap v1432: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:09.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:08 smithi094 ceph-mon[40484]: pgmap v1432: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:10.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:10 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:10 smithi094 ceph-mon[29291]: pgmap v1433: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:39:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:39:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:39:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:10 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:10 smithi094 ceph-mon[40484]: pgmap v1433: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:11.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:10 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:11.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:10 smithi151 ceph-mon[32923]: pgmap v1433: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:13.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:12 smithi094 ceph-mon[40484]: pgmap v1434: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:13.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:12 smithi094 ceph-mon[29291]: pgmap v1434: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:13.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:12 smithi151 ceph-mon[32923]: pgmap v1434: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:15.017 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:14 smithi151 ceph-mon[32923]: pgmap v1435: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:15.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:14 smithi094 ceph-mon[29291]: pgmap v1435: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:15.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:14 smithi094 ceph-mon[40484]: pgmap v1435: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:17.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:16 smithi094 ceph-mon[29291]: pgmap v1436: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:17.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:16 smithi094 ceph-mon[40484]: pgmap v1436: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:17.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:16 smithi151 ceph-mon[32923]: pgmap v1436: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:19.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:18 smithi094 ceph-mon[29291]: pgmap v1437: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:19.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:18 smithi094 ceph-mon[40484]: pgmap v1437: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:19.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:18 smithi151 ceph-mon[32923]: pgmap v1437: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:19.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:39:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:39:20.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:39:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:39:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:39:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:20 smithi094 ceph-mon[29291]: pgmap v1438: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:20 smithi094 ceph-mon[40484]: pgmap v1438: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:21.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:21.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:20 smithi151 ceph-mon[32923]: pgmap v1438: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:21.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:23.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:22 smithi094 ceph-mon[40484]: pgmap v1439: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:23.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:22 smithi094 ceph-mon[29291]: pgmap v1439: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:23.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:22 smithi151 ceph-mon[32923]: pgmap v1439: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:25.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:24 smithi094 ceph-mon[29291]: pgmap v1440: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:24 smithi094 ceph-mon[40484]: pgmap v1440: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:25.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:24 smithi151 ceph-mon[32923]: pgmap v1440: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:27.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:26 smithi094 ceph-mon[40484]: pgmap v1441: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:27.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:26 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:39:27.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:26 smithi094 ceph-mon[29291]: pgmap v1441: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:27.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:26 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:39:27.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:26 smithi151 ceph-mon[32923]: pgmap v1441: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:27.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:26 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:39:28.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:28 smithi094 ceph-mon[29291]: pgmap v1442: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:28 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:39:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:28 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:39:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:28 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:28 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:39:28.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:28 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:39:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:28 smithi094 ceph-mon[40484]: pgmap v1442: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:28 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:39:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:28 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:39:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:28 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:28 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:39:28.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:28 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:39:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:28 smithi151 ceph-mon[32923]: pgmap v1442: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:28 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:39:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:28 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:39:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:28 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:28 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:39:28.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:28 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:39:29.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:39:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:39:30.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:30.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:30 smithi094 ceph-mon[40484]: pgmap v1443: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:30.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:30.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:30 smithi094 ceph-mon[29291]: pgmap v1443: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:30.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:30.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:30 smithi151 ceph-mon[32923]: pgmap v1443: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:30.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:39:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:39:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:39:32.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:32 smithi094 ceph-mon[29291]: pgmap v1444: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:32.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:32 smithi094 ceph-mon[40484]: pgmap v1444: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:32.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:32 smithi151 ceph-mon[32923]: pgmap v1444: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:34.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:34 smithi094 ceph-mon[29291]: pgmap v1445: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:34.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:34 smithi094 ceph-mon[40484]: pgmap v1445: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:34.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:34 smithi151 ceph-mon[32923]: pgmap v1445: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:35.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:35.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:35.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:36.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:36 smithi094 ceph-mon[29291]: pgmap v1446: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:36.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:36 smithi094 ceph-mon[40484]: pgmap v1446: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:36.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:36 smithi151 ceph-mon[32923]: pgmap v1446: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:38.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:38 smithi094 ceph-mon[29291]: pgmap v1447: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:38.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:38 smithi094 ceph-mon[40484]: pgmap v1447: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:38.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:38 smithi151 ceph-mon[32923]: pgmap v1447: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:39.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:39:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:39:40.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:40.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:40 smithi094 ceph-mon[40484]: pgmap v1448: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:40.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:40.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:40 smithi094 ceph-mon[29291]: pgmap v1448: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:40.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:40 smithi151 ceph-mon[32923]: pgmap v1448: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:40.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:39:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:39:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:39:42.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:42 smithi094 ceph-mon[29291]: pgmap v1449: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:42.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:42 smithi094 ceph-mon[40484]: pgmap v1449: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:42.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:42 smithi151 ceph-mon[32923]: pgmap v1449: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:44.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:44 smithi094 ceph-mon[29291]: pgmap v1450: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:44.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:44 smithi094 ceph-mon[40484]: pgmap v1450: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:44.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:44 smithi151 ceph-mon[32923]: pgmap v1450: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:46.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:46 smithi094 ceph-mon[40484]: pgmap v1451: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:46.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:46 smithi094 ceph-mon[29291]: pgmap v1451: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:46.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:46 smithi151 ceph-mon[32923]: pgmap v1451: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:48.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:48 smithi094 ceph-mon[40484]: pgmap v1452: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:48.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:48 smithi094 ceph-mon[29291]: pgmap v1452: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:48.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:48 smithi151 ceph-mon[32923]: pgmap v1452: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:49.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:39:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:39:50.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:50.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:50 smithi094 ceph-mon[40484]: pgmap v1453: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:50.532 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:50.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:50.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:50 smithi094 ceph-mon[29291]: pgmap v1453: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:50.532 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:50.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:50 smithi151 ceph-mon[32923]: pgmap v1453: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:50.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:50.961 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:39:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:39:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:39:52.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:52 smithi094 ceph-mon[29291]: pgmap v1454: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:52.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:52 smithi094 ceph-mon[40484]: pgmap v1454: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:52.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:52 smithi151 ceph-mon[32923]: pgmap v1454: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:54.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:54 smithi094 ceph-mon[29291]: pgmap v1455: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:54.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:54 smithi094 ceph-mon[40484]: pgmap v1455: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:54.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:54 smithi151 ceph-mon[32923]: pgmap v1455: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:56.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:56 smithi094 ceph-mon[29291]: pgmap v1456: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:56.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:56 smithi094 ceph-mon[40484]: pgmap v1456: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:56.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:56 smithi151 ceph-mon[32923]: pgmap v1456: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:39:58.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:39:58 smithi094 ceph-mon[29291]: pgmap v1457: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:58.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:39:58 smithi094 ceph-mon[40484]: pgmap v1457: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:58.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:39:58 smithi151 ceph-mon[32923]: pgmap v1457: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:59.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:39:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:40:00.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:40:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:40:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[40484]: pgmap v1458: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[40484]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-12T03:40:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[40484]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-12T03:40:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[40484]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-12T03:40:00.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[40484]: application not enabled on pool 'ListObjectssmithi094-79253-1' 2024-03-12T03:40:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[40484]: application not enabled on pool 'SetOpFlagssmithi094-79449-1' 2024-03-12T03:40:00.711 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[40484]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-12T03:40:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[29291]: pgmap v1458: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[29291]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-12T03:40:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[29291]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-12T03:40:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[29291]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-12T03:40:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[29291]: application not enabled on pool 'ListObjectssmithi094-79253-1' 2024-03-12T03:40:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[29291]: application not enabled on pool 'SetOpFlagssmithi094-79449-1' 2024-03-12T03:40:00.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:00 smithi094 ceph-mon[29291]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-12T03:40:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:00 smithi151 ceph-mon[32923]: pgmap v1458: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:00 smithi151 ceph-mon[32923]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-12T03:40:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:00 smithi151 ceph-mon[32923]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-12T03:40:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:00 smithi151 ceph-mon[32923]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-12T03:40:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:00 smithi151 ceph-mon[32923]: application not enabled on pool 'ListObjectssmithi094-79253-1' 2024-03-12T03:40:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:00 smithi151 ceph-mon[32923]: application not enabled on pool 'SetOpFlagssmithi094-79449-1' 2024-03-12T03:40:00.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:00 smithi151 ceph-mon[32923]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-12T03:40:02.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:02 smithi094 ceph-mon[29291]: pgmap v1459: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:02.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:02 smithi094 ceph-mon[40484]: pgmap v1459: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:02.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:02 smithi151 ceph-mon[32923]: pgmap v1459: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:04.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:04 smithi094 ceph-mon[29291]: pgmap v1460: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:04.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:04 smithi094 ceph-mon[40484]: pgmap v1460: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:04 smithi151 ceph-mon[32923]: pgmap v1460: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:05.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:05.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:06.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:06 smithi094 ceph-mon[29291]: pgmap v1461: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:06.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:06 smithi094 ceph-mon[40484]: pgmap v1461: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:06.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:06 smithi151 ceph-mon[32923]: pgmap v1461: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:08.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:08 smithi094 ceph-mon[29291]: pgmap v1462: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:08 smithi094 ceph-mon[40484]: pgmap v1462: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:08.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:08 smithi151 ceph-mon[32923]: pgmap v1462: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:09.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:40:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:40:10.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:10 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:10.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:10 smithi094 ceph-mon[29291]: pgmap v1463: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:10.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:40:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:40:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:40:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:10 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:10 smithi094 ceph-mon[40484]: pgmap v1463: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:10 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:10 smithi151 ceph-mon[32923]: pgmap v1463: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:12 smithi094 ceph-mon[29291]: pgmap v1464: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:12.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:12 smithi094 ceph-mon[40484]: pgmap v1464: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:12 smithi151 ceph-mon[32923]: pgmap v1464: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:14.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:14 smithi094 ceph-mon[29291]: pgmap v1465: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:14 smithi094 ceph-mon[40484]: pgmap v1465: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:14 smithi151 ceph-mon[32923]: pgmap v1465: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:16.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:16 smithi094 ceph-mon[29291]: pgmap v1466: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:16.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:16 smithi094 ceph-mon[40484]: pgmap v1466: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:16.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:16 smithi151 ceph-mon[32923]: pgmap v1466: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:18.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:18 smithi094 ceph-mon[29291]: pgmap v1467: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:18 smithi094 ceph-mon[40484]: pgmap v1467: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:18 smithi151 ceph-mon[32923]: pgmap v1467: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:19.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:40:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:40:20.709 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:40:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:40:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:40:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:20 smithi094 ceph-mon[40484]: pgmap v1468: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:20.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:20 smithi094 ceph-mon[29291]: pgmap v1468: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:20.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:20 smithi151 ceph-mon[32923]: pgmap v1468: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:22.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:22 smithi094 ceph-mon[40484]: pgmap v1469: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:22.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:22 smithi094 ceph-mon[29291]: pgmap v1469: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:22.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:22 smithi151 ceph-mon[32923]: pgmap v1469: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:24.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:24 smithi094 ceph-mon[29291]: pgmap v1470: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:24.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:24 smithi094 ceph-mon[40484]: pgmap v1470: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:24.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:24 smithi151 ceph-mon[32923]: pgmap v1470: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:26.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:26 smithi094 ceph-mon[29291]: pgmap v1471: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:26.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:26 smithi094 ceph-mon[40484]: pgmap v1471: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:26.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:26 smithi151 ceph-mon[32923]: pgmap v1471: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:28.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:28 smithi094 ceph-mon[29291]: pgmap v1472: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:28.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:28 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:40:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:28 smithi094 ceph-mon[40484]: pgmap v1472: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:28.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:28 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:40:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:28 smithi151 ceph-mon[32923]: pgmap v1472: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:28.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:28 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:40:29.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:40:28 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:40:29.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:40:29.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:40:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:29 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:40:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:29 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:40:29.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:29 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:40:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:40:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:40:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:29 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:40:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:29 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:40:29.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:29 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:40:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:40:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:40:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:29 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:40:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:29 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:40:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:29 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:40:30.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:40:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:40:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:40:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:30 smithi094 ceph-mon[40484]: pgmap v1473: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:30 smithi094 ceph-mon[29291]: pgmap v1473: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:30 smithi151 ceph-mon[32923]: pgmap v1473: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:32.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:32 smithi094 ceph-mon[29291]: pgmap v1474: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:32.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:32 smithi094 ceph-mon[40484]: pgmap v1474: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:32 smithi151 ceph-mon[32923]: pgmap v1474: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:34.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:34 smithi094 ceph-mon[29291]: pgmap v1475: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:34.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:34 smithi094 ceph-mon[40484]: pgmap v1475: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:34 smithi151 ceph-mon[32923]: pgmap v1475: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:35.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:36.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:36 smithi094 ceph-mon[29291]: pgmap v1476: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:36 smithi094 ceph-mon[40484]: pgmap v1476: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:36 smithi151 ceph-mon[32923]: pgmap v1476: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:38 smithi094 ceph-mon[29291]: pgmap v1477: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:38 smithi094 ceph-mon[40484]: pgmap v1477: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:38.961 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:38 smithi151 ceph-mon[32923]: pgmap v1477: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:39.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:40:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:40:40.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:40.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:40 smithi094 ceph-mon[29291]: pgmap v1478: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:40:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:40:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:40:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:40 smithi094 ceph-mon[40484]: pgmap v1478: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:40 smithi151 ceph-mon[32923]: pgmap v1478: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:42.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:42 smithi094 ceph-mon[29291]: pgmap v1479: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:42 smithi094 ceph-mon[40484]: pgmap v1479: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:42 smithi151 ceph-mon[32923]: pgmap v1479: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:44.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:44 smithi094 ceph-mon[29291]: pgmap v1480: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:44.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:44 smithi094 ceph-mon[40484]: pgmap v1480: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:44 smithi151 ceph-mon[32923]: pgmap v1480: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:46.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:46 smithi094 ceph-mon[29291]: pgmap v1481: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:46 smithi094 ceph-mon[40484]: pgmap v1481: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:46 smithi151 ceph-mon[32923]: pgmap v1481: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:48.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:48 smithi094 ceph-mon[29291]: pgmap v1482: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:48.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:48 smithi094 ceph-mon[40484]: pgmap v1482: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:48 smithi151 ceph-mon[32923]: pgmap v1482: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:49.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:40:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:40:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:50 smithi094 ceph-mon[29291]: pgmap v1483: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:50.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:40:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:40:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:40:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:50 smithi094 ceph-mon[40484]: pgmap v1483: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:50 smithi151 ceph-mon[32923]: pgmap v1483: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:52.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:52 smithi094 ceph-mon[40484]: pgmap v1484: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:52.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:52 smithi094 ceph-mon[29291]: pgmap v1484: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:52.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:52 smithi151 ceph-mon[32923]: pgmap v1484: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:54.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:54 smithi094 ceph-mon[29291]: pgmap v1485: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:54.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:54 smithi094 ceph-mon[40484]: pgmap v1485: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:54 smithi151 ceph-mon[32923]: pgmap v1485: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:56.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:56 smithi094 ceph-mon[29291]: pgmap v1486: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:56.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:56 smithi094 ceph-mon[40484]: pgmap v1486: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:56.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:56 smithi151 ceph-mon[32923]: pgmap v1486: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:40:58.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:40:58 smithi094 ceph-mon[29291]: pgmap v1487: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:40:58 smithi094 ceph-mon[40484]: pgmap v1487: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:40:58 smithi151 ceph-mon[32923]: pgmap v1487: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:59.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:40:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:41:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:00 smithi094 ceph-mon[29291]: pgmap v1488: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:00.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:41:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:41:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:41:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:00 smithi094 ceph-mon[40484]: pgmap v1488: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:00 smithi151 ceph-mon[32923]: pgmap v1488: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:02.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:02 smithi094 ceph-mon[29291]: pgmap v1489: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:02 smithi094 ceph-mon[40484]: pgmap v1489: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:02 smithi151 ceph-mon[32923]: pgmap v1489: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:04 smithi094 ceph-mon[29291]: pgmap v1490: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:04 smithi094 ceph-mon[40484]: pgmap v1490: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:04 smithi151 ceph-mon[32923]: pgmap v1490: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:05.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:06 smithi094 ceph-mon[29291]: pgmap v1491: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:06 smithi094 ceph-mon[40484]: pgmap v1491: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:06 smithi151 ceph-mon[32923]: pgmap v1491: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:08 smithi094 ceph-mon[29291]: pgmap v1492: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:08 smithi094 ceph-mon[40484]: pgmap v1492: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:08 smithi151 ceph-mon[32923]: pgmap v1492: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:09.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:41:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:41:10.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:10 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:10 smithi094 ceph-mon[29291]: pgmap v1493: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:10 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:10 smithi094 ceph-mon[40484]: pgmap v1493: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:41:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:41:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:41:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:10 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:10 smithi151 ceph-mon[32923]: pgmap v1493: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:12.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:12 smithi094 ceph-mon[29291]: pgmap v1494: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:12 smithi094 ceph-mon[40484]: pgmap v1494: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:12 smithi151 ceph-mon[32923]: pgmap v1494: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:14.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:14 smithi094 ceph-mon[29291]: pgmap v1495: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:14 smithi094 ceph-mon[40484]: pgmap v1495: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:14 smithi151 ceph-mon[32923]: pgmap v1495: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:16.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:16 smithi094 ceph-mon[29291]: pgmap v1496: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:16 smithi094 ceph-mon[40484]: pgmap v1496: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:16 smithi151 ceph-mon[32923]: pgmap v1496: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:18.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:18 smithi094 ceph-mon[29291]: pgmap v1497: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:18 smithi094 ceph-mon[40484]: pgmap v1497: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:18 smithi151 ceph-mon[32923]: pgmap v1497: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:19.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:41:18 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:41:20.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:41:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:41:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:41:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:20 smithi094 ceph-mon[29291]: pgmap v1498: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:20.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:20.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:20 smithi094 ceph-mon[40484]: pgmap v1498: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:20.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:20 smithi151 ceph-mon[32923]: pgmap v1498: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:22 smithi094 ceph-mon[29291]: pgmap v1499: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:22 smithi094 ceph-mon[40484]: pgmap v1499: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:22 smithi151 ceph-mon[32923]: pgmap v1499: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:24.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:24 smithi094 ceph-mon[29291]: pgmap v1500: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:24.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:24 smithi094 ceph-mon[40484]: pgmap v1500: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:24 smithi151 ceph-mon[32923]: pgmap v1500: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:26.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:26 smithi094 ceph-mon[29291]: pgmap v1501: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:26 smithi094 ceph-mon[40484]: pgmap v1501: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:26 smithi151 ceph-mon[32923]: pgmap v1501: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:28.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:28 smithi094 ceph-mon[29291]: pgmap v1502: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:28 smithi094 ceph-mon[40484]: pgmap v1502: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:28 smithi151 ceph-mon[32923]: pgmap v1502: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:29.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:41:29 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:41:29.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:29 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:41:29.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:29 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:41:29.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:29 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:41:30.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:41:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:41:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:41:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:30 smithi094 ceph-mon[29291]: pgmap v1503: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:30 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:41:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:30 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:41:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:30 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:41:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:30 smithi094 ceph-mon[40484]: pgmap v1503: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:30 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:41:30.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:30 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:41:30.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:30 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:41:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:30 smithi151 ceph-mon[32923]: pgmap v1503: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:30 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:41:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:30 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:41:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:30 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:41:32.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:32 smithi094 ceph-mon[40484]: pgmap v1504: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:32 smithi094 ceph-mon[29291]: pgmap v1504: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:32 smithi151 ceph-mon[32923]: pgmap v1504: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:34.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:34 smithi094 ceph-mon[29291]: pgmap v1505: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:34 smithi094 ceph-mon[40484]: pgmap v1505: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:34 smithi151 ceph-mon[32923]: pgmap v1505: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:35.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:36.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:36 smithi094 ceph-mon[29291]: pgmap v1506: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:36 smithi094 ceph-mon[40484]: pgmap v1506: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:36 smithi151 ceph-mon[32923]: pgmap v1506: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:38 smithi094 ceph-mon[29291]: pgmap v1507: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:38 smithi094 ceph-mon[40484]: pgmap v1507: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:38 smithi151 ceph-mon[32923]: pgmap v1507: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:39.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:41:39 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:41:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:40 smithi094 ceph-mon[40484]: pgmap v1508: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:40 smithi094 ceph-mon[29291]: pgmap v1508: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:40.961 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:41:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:41:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:41:40.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:40.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:40 smithi151 ceph-mon[32923]: pgmap v1508: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:42.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:42 smithi094 ceph-mon[29291]: pgmap v1509: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:42 smithi094 ceph-mon[40484]: pgmap v1509: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:42 smithi151 ceph-mon[32923]: pgmap v1509: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:44 smithi094 ceph-mon[29291]: pgmap v1510: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:44 smithi094 ceph-mon[40484]: pgmap v1510: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:44 smithi151 ceph-mon[32923]: pgmap v1510: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:46.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:46 smithi094 ceph-mon[29291]: pgmap v1511: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:46 smithi094 ceph-mon[40484]: pgmap v1511: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:46 smithi151 ceph-mon[32923]: pgmap v1511: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:48 smithi094 ceph-mon[29291]: pgmap v1512: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:48.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:48 smithi094 ceph-mon[40484]: pgmap v1512: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:48 smithi151 ceph-mon[32923]: pgmap v1512: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:49.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:41:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:41:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:50 smithi094 ceph-mon[29291]: pgmap v1513: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:50.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:41:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:41:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:41:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:50.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:50 smithi094 ceph-mon[40484]: pgmap v1513: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:50.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:50 smithi151 ceph-mon[32923]: pgmap v1513: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:52.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:52 smithi094 ceph-mon[29291]: pgmap v1514: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:52 smithi094 ceph-mon[40484]: pgmap v1514: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:52.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:52 smithi151 ceph-mon[32923]: pgmap v1514: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:54.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:54 smithi094 ceph-mon[29291]: pgmap v1515: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:54.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:54 smithi094 ceph-mon[40484]: pgmap v1515: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:54 smithi151 ceph-mon[32923]: pgmap v1515: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:56.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:56 smithi094 ceph-mon[29291]: pgmap v1516: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:56.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:56 smithi094 ceph-mon[40484]: pgmap v1516: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:56.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:56 smithi151 ceph-mon[32923]: pgmap v1516: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:41:58.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:41:58 smithi094 ceph-mon[29291]: pgmap v1517: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:41:58 smithi094 ceph-mon[40484]: pgmap v1517: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:41:58 smithi151 ceph-mon[32923]: pgmap v1517: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:59.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:41:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:42:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:00 smithi094 ceph-mon[29291]: pgmap v1518: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:00.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:42:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:42:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:42:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:00 smithi094 ceph-mon[40484]: pgmap v1518: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:00 smithi151 ceph-mon[32923]: pgmap v1518: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:02.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:02 smithi094 ceph-mon[29291]: pgmap v1519: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:02 smithi094 ceph-mon[40484]: pgmap v1519: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:02 smithi151 ceph-mon[32923]: pgmap v1519: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:04 smithi094 ceph-mon[29291]: pgmap v1520: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:04 smithi094 ceph-mon[40484]: pgmap v1520: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:04 smithi151 ceph-mon[32923]: pgmap v1520: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:05.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:06 smithi094 ceph-mon[29291]: pgmap v1521: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:06 smithi094 ceph-mon[40484]: pgmap v1521: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:06 smithi151 ceph-mon[32923]: pgmap v1521: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:08 smithi094 ceph-mon[29291]: pgmap v1522: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:08 smithi094 ceph-mon[40484]: pgmap v1522: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:08 smithi151 ceph-mon[32923]: pgmap v1522: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:09.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:42:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:42:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:10 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:10 smithi094 ceph-mon[29291]: pgmap v1523: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:10.961 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:42:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:42:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:42:10.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:10 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:10.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:10 smithi094 ceph-mon[40484]: pgmap v1523: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:10 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:10 smithi151 ceph-mon[32923]: pgmap v1523: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:12.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:12 smithi094 ceph-mon[29291]: pgmap v1524: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:12 smithi094 ceph-mon[40484]: pgmap v1524: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:12 smithi151 ceph-mon[32923]: pgmap v1524: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:14.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:14 smithi094 ceph-mon[29291]: pgmap v1525: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:14 smithi094 ceph-mon[40484]: pgmap v1525: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:14 smithi151 ceph-mon[32923]: pgmap v1525: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:16.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:16 smithi094 ceph-mon[29291]: pgmap v1526: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:16 smithi094 ceph-mon[40484]: pgmap v1526: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:16 smithi151 ceph-mon[32923]: pgmap v1526: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:18.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:18 smithi094 ceph-mon[29291]: pgmap v1527: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:18 smithi094 ceph-mon[40484]: pgmap v1527: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:18 smithi151 ceph-mon[32923]: pgmap v1527: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:19.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:42:19 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:42:20.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:20 smithi094 ceph-mon[40484]: pgmap v1528: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:20 smithi094 ceph-mon[29291]: pgmap v1528: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:20.961 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:42:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:42:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:42:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:20 smithi151 ceph-mon[32923]: pgmap v1528: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:22.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:22 smithi094 ceph-mon[29291]: pgmap v1529: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:22 smithi094 ceph-mon[40484]: pgmap v1529: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:22 smithi151 ceph-mon[32923]: pgmap v1529: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:24.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:24 smithi094 ceph-mon[29291]: pgmap v1530: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:24.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:24 smithi094 ceph-mon[40484]: pgmap v1530: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:24 smithi151 ceph-mon[32923]: pgmap v1530: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:26.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:26 smithi094 ceph-mon[29291]: pgmap v1531: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:26 smithi094 ceph-mon[40484]: pgmap v1531: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:26 smithi151 ceph-mon[32923]: pgmap v1531: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:28 smithi094 ceph-mon[29291]: pgmap v1532: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:28 smithi094 ceph-mon[40484]: pgmap v1532: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:28 smithi151 ceph-mon[32923]: pgmap v1532: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:29.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:42:29 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:42:30.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:30.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:30 smithi094 ceph-mon[29291]: pgmap v1533: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:30.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:30 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:42:30.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:42:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:42:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:42:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:30 smithi094 ceph-mon[40484]: pgmap v1533: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:30 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:42:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:30 smithi151 ceph-mon[32923]: pgmap v1533: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:30 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:42:32.821 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:32 smithi094 ceph-mon[29291]: pgmap v1534: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:32.821 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:32 smithi094 ceph-mon[40484]: pgmap v1534: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:32 smithi151 ceph-mon[32923]: pgmap v1534: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:34.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:34 smithi094 ceph-mon[29291]: pgmap v1535: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:34 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:42:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:34 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:42:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:42:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:42:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:34 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:42:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:34 smithi094 ceph-mon[40484]: pgmap v1535: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:34 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:42:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:34 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:42:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:42:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:42:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:34 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:42:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:34 smithi151 ceph-mon[32923]: pgmap v1535: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:34 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:42:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:34 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:42:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:42:34.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:42:34.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:34 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:42:35.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:36.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:36 smithi094 ceph-mon[29291]: pgmap v1536: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:36 smithi094 ceph-mon[40484]: pgmap v1536: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:36 smithi151 ceph-mon[32923]: pgmap v1536: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:42:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:38 smithi094 ceph-mon[29291]: pgmap v1537: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:38 smithi094 ceph-mon[40484]: pgmap v1537: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:38 smithi151 ceph-mon[32923]: pgmap v1537: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:39.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:42:39 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:42:40.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:40 smithi094 ceph-mon[40484]: pgmap v1538: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:40 smithi094 ceph-mon[29291]: pgmap v1538: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:42:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:42:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:42:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:40.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:40 smithi151 ceph-mon[32923]: pgmap v1538: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:42 smithi151 ceph-mon[32923]: pgmap v1539: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:42 smithi094 ceph-mon[29291]: pgmap v1539: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:42 smithi094 ceph-mon[40484]: pgmap v1539: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:45.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:44 smithi094 ceph-mon[29291]: pgmap v1540: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:45.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:44 smithi094 ceph-mon[40484]: pgmap v1540: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:45.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:44 smithi151 ceph-mon[32923]: pgmap v1540: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:47.017 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:46 smithi151 ceph-mon[32923]: pgmap v1541: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:47.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:46 smithi094 ceph-mon[29291]: pgmap v1541: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:46 smithi094 ceph-mon[40484]: pgmap v1541: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:49.078 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:48 smithi151 ceph-mon[32923]: pgmap v1542: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:49.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:48 smithi094 ceph-mon[29291]: pgmap v1542: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:48 smithi094 ceph-mon[40484]: pgmap v1542: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:49.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:42:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:42:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:50 smithi094 ceph-mon[29291]: pgmap v1543: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:50.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:42:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:42:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:42:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:50 smithi094 ceph-mon[40484]: pgmap v1543: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:51.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:50 smithi151 ceph-mon[32923]: pgmap v1543: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:51.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:53.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:52 smithi094 ceph-mon[29291]: pgmap v1544: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:53.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:52 smithi094 ceph-mon[40484]: pgmap v1544: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:53.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:52 smithi151 ceph-mon[32923]: pgmap v1544: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:55.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:54 smithi094 ceph-mon[40484]: pgmap v1545: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:55.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:54 smithi094 ceph-mon[29291]: pgmap v1545: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:55.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:54 smithi151 ceph-mon[32923]: pgmap v1545: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:57.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:56 smithi094 ceph-mon[40484]: pgmap v1546: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:57.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:56 smithi094 ceph-mon[29291]: pgmap v1546: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:57.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:56 smithi151 ceph-mon[32923]: pgmap v1546: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:59.090 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:42:58 smithi151 ceph-mon[32923]: pgmap v1547: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:59.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:42:58 smithi094 ceph-mon[40484]: pgmap v1547: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:59.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:42:58 smithi094 ceph-mon[29291]: pgmap v1547: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:59.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:42:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:43:00.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:00.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:00 smithi094 ceph-mon[40484]: pgmap v1548: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:00 smithi094 ceph-mon[29291]: pgmap v1548: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:00.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:43:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:43:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:43:01.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:01.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:00 smithi151 ceph-mon[32923]: pgmap v1548: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:03.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:02 smithi094 ceph-mon[29291]: pgmap v1549: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:03.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:02 smithi094 ceph-mon[40484]: pgmap v1549: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:03.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:02 smithi151 ceph-mon[32923]: pgmap v1549: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:05.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:04 smithi094 ceph-mon[29291]: pgmap v1550: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:05.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:04 smithi094 ceph-mon[40484]: pgmap v1550: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:05.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:04 smithi151 ceph-mon[32923]: pgmap v1550: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:06.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:06.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:06.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:07.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:06 smithi094 ceph-mon[29291]: pgmap v1551: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:07.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:06 smithi094 ceph-mon[40484]: pgmap v1551: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:07.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:06 smithi151 ceph-mon[32923]: pgmap v1551: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:09.101 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:08 smithi151 ceph-mon[32923]: pgmap v1552: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:09.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:08 smithi094 ceph-mon[40484]: pgmap v1552: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:09.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:08 smithi094 ceph-mon[29291]: pgmap v1552: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:09.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:43:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:43:10.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:10 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:10 smithi094 ceph-mon[29291]: pgmap v1553: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:43:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:43:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:43:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:10 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:10 smithi094 ceph-mon[40484]: pgmap v1553: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:11.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:10 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:11.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:10 smithi151 ceph-mon[32923]: pgmap v1553: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:13.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:12 smithi094 ceph-mon[29291]: pgmap v1554: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:13.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:12 smithi094 ceph-mon[40484]: pgmap v1554: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:13.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:12 smithi151 ceph-mon[32923]: pgmap v1554: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:15.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:14 smithi094 ceph-mon[40484]: pgmap v1555: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:15.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:14 smithi094 ceph-mon[29291]: pgmap v1555: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:15.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:14 smithi151 ceph-mon[32923]: pgmap v1555: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:17.017 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:16 smithi151 ceph-mon[32923]: pgmap v1556: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:17.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:16 smithi094 ceph-mon[40484]: pgmap v1556: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:17.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:16 smithi094 ceph-mon[29291]: pgmap v1556: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:19.112 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:18 smithi151 ceph-mon[32923]: pgmap v1557: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:19.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:18 smithi094 ceph-mon[29291]: pgmap v1557: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:19.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:18 smithi094 ceph-mon[40484]: pgmap v1557: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:19.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:43:19 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:43:20.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:20 smithi094 ceph-mon[40484]: pgmap v1558: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:20 smithi094 ceph-mon[29291]: pgmap v1558: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:20.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:43:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:43:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:43:21.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:20 smithi151 ceph-mon[32923]: pgmap v1558: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:21.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:21.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:23.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:22 smithi094 ceph-mon[29291]: pgmap v1559: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:23.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:22 smithi094 ceph-mon[40484]: pgmap v1559: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:23.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:22 smithi151 ceph-mon[32923]: pgmap v1559: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:25.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:24 smithi094 ceph-mon[29291]: pgmap v1560: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:24 smithi094 ceph-mon[40484]: pgmap v1560: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:25.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:24 smithi151 ceph-mon[32923]: pgmap v1560: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:27.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:26 smithi094 ceph-mon[29291]: pgmap v1561: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:27.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:26 smithi094 ceph-mon[40484]: pgmap v1561: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:27.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:26 smithi151 ceph-mon[32923]: pgmap v1561: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:29.123 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:28 smithi151 ceph-mon[32923]: pgmap v1562: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:29.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:28 smithi094 ceph-mon[29291]: pgmap v1562: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:28 smithi094 ceph-mon[40484]: pgmap v1562: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:29.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:43:29 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:43:30.789 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:43:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:43:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:43:31.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:30 smithi094 ceph-mon[40484]: pgmap v1563: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:31.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:30 smithi094 ceph-mon[29291]: pgmap v1563: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:31.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:31.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:30 smithi151 ceph-mon[32923]: pgmap v1563: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:31.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:33.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:32 smithi094 ceph-mon[29291]: pgmap v1564: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:33.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:32 smithi094 ceph-mon[40484]: pgmap v1564: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:33.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:32 smithi151 ceph-mon[32923]: pgmap v1564: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:35.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:34 smithi094 ceph-mon[40484]: pgmap v1565: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:35.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:34 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:43:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:34 smithi094 ceph-mon[29291]: pgmap v1565: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:35.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:34 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:43:35.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:34 smithi151 ceph-mon[32923]: pgmap v1565: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:35.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:34 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:43:36.054 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:36.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:37.083 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:36 smithi151 ceph-mon[32923]: pgmap v1566: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:37.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:36 smithi094 ceph-mon[29291]: pgmap v1566: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:36 smithi094 ceph-mon[40484]: pgmap v1566: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:38 smithi151 ceph-mon[32923]: pgmap v1567: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:38 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:43:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:38 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:43:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:38 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:43:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:38 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:43:38.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:38 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:43:39.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:38 smithi094 ceph-mon[29291]: pgmap v1567: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:38 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:43:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:38 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:43:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:38 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:43:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:38 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:43:39.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:38 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:43:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:38 smithi094 ceph-mon[40484]: pgmap v1567: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:38 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:43:39.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:38 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:43:39.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:38 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:43:39.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:38 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:43:39.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:38 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:43:39.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:43:39 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:43:40.805 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:43:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:43:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:43:41.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:40 smithi094 ceph-mon[40484]: pgmap v1568: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:41.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:41.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:40 smithi094 ceph-mon[29291]: pgmap v1568: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:41.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:40 smithi151 ceph-mon[32923]: pgmap v1568: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:43.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:42 smithi094 ceph-mon[29291]: pgmap v1569: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:43.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:42 smithi094 ceph-mon[40484]: pgmap v1569: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:42 smithi151 ceph-mon[32923]: pgmap v1569: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:44.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:43 smithi094 ceph-mon[29291]: pgmap v1570: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:44.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:43 smithi094 ceph-mon[40484]: pgmap v1570: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:44.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:43 smithi151 ceph-mon[32923]: pgmap v1570: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:46.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:46 smithi094 ceph-mon[29291]: pgmap v1571: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:46.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:46 smithi094 ceph-mon[40484]: pgmap v1571: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:46.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:46 smithi151 ceph-mon[32923]: pgmap v1571: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:43:48.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:48 smithi094 ceph-mon[29291]: pgmap v1572: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:48.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:48 smithi094 ceph-mon[40484]: pgmap v1572: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:48.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:48 smithi151 ceph-mon[32923]: pgmap v1572: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:49.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:43:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:43:50.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:50 smithi094 ceph-mon[40484]: pgmap v1573: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:50.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:50.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:50 smithi094 ceph-mon[29291]: pgmap v1573: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:50.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:50.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:50 smithi151 ceph-mon[32923]: pgmap v1573: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:50.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:50.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:43:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:43:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:43:51.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:51 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:51.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:51 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:51.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:51 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:52.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:52 smithi094 ceph-mon[29291]: pgmap v1574: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:52.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:52 smithi094 ceph-mon[40484]: pgmap v1574: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:52.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:52 smithi151 ceph-mon[32923]: pgmap v1574: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:54.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:54 smithi094 ceph-mon[29291]: pgmap v1575: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:54.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:54 smithi094 ceph-mon[40484]: pgmap v1575: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:54.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:54 smithi151 ceph-mon[32923]: pgmap v1575: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:56.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:56 smithi094 ceph-mon[29291]: pgmap v1576: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:56.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:56 smithi094 ceph-mon[40484]: pgmap v1576: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:56.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:56 smithi151 ceph-mon[32923]: pgmap v1576: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:58.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:43:58 smithi094 ceph-mon[29291]: pgmap v1577: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:58.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:43:58 smithi094 ceph-mon[40484]: pgmap v1577: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:58.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:43:58 smithi151 ceph-mon[32923]: pgmap v1577: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:43:59.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:43:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:44:00.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:00 smithi094 ceph-mon[29291]: pgmap v1578: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:44:00.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:00.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:00 smithi094 ceph-mon[40484]: pgmap v1578: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:44:00.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:00.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:00 smithi151 ceph-mon[32923]: pgmap v1578: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:44:00.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:00.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:44:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:44:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:44:02.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:02 smithi094 ceph-mon[29291]: pgmap v1579: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:02.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:02 smithi094 ceph-mon[40484]: pgmap v1579: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:02.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:02 smithi151 ceph-mon[32923]: pgmap v1579: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:04.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:04 smithi094 ceph-mon[29291]: pgmap v1580: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:04.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:04 smithi094 ceph-mon[40484]: pgmap v1580: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:04.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:04 smithi151 ceph-mon[32923]: pgmap v1580: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:06.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:06 smithi094 ceph-mon[29291]: pgmap v1581: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:06.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:06 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:06.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:06 smithi094 ceph-mon[40484]: pgmap v1581: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:06.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:06 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:06.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:06 smithi151 ceph-mon[32923]: pgmap v1581: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:06.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:06 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:08.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:08 smithi151 ceph-mon[32923]: pgmap v1582: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:08.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:08 smithi094 ceph-mon[29291]: pgmap v1582: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:08.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:08 smithi094 ceph-mon[40484]: pgmap v1582: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:09.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:44:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:44:10.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:10 smithi094 ceph-mon[29291]: pgmap v1583: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:10.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:10 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:10.710 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:44:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:44:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:44:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:10 smithi094 ceph-mon[40484]: pgmap v1583: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:10.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:10 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:10 smithi151 ceph-mon[32923]: pgmap v1583: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:10.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:10 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:12.709 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:12 smithi094 ceph-mon[40484]: pgmap v1584: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:12.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:12 smithi094 ceph-mon[29291]: pgmap v1584: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:12.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:12 smithi151 ceph-mon[32923]: pgmap v1584: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:14.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:14 smithi094 ceph-mon[29291]: pgmap v1585: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:14 smithi094 ceph-mon[40484]: pgmap v1585: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:14 smithi151 ceph-mon[32923]: pgmap v1585: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:16.717 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:16 smithi151 ceph-mon[32923]: pgmap v1586: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:44:16.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:16 smithi094 ceph-mon[29291]: pgmap v1586: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:44:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:16 smithi094 ceph-mon[40484]: pgmap v1586: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:44:18.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:18 smithi151 ceph-mon[32923]: pgmap v1587: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:44:18.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:18 smithi094 ceph-mon[29291]: pgmap v1587: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:44:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:18 smithi094 ceph-mon[40484]: pgmap v1587: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:44:19.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:44:19 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:44:20.762 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:44:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:44:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:44:20.762 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:20 smithi094 ceph-mon[40484]: pgmap v1588: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:20.762 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:20.763 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:20.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:20 smithi094 ceph-mon[29291]: pgmap v1588: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:20.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:20.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:20 smithi151 ceph-mon[32923]: pgmap v1588: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:20.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:20.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:22.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:22 smithi094 ceph-mon[29291]: pgmap v1589: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:44:22.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:22 smithi094 ceph-mon[40484]: pgmap v1589: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:44:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:22 smithi151 ceph-mon[32923]: pgmap v1589: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:44:24.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:24 smithi094 ceph-mon[29291]: pgmap v1590: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:24.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:24 smithi094 ceph-mon[40484]: pgmap v1590: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:24 smithi151 ceph-mon[32923]: pgmap v1590: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:26.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:26 smithi094 ceph-mon[29291]: pgmap v1591: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:26 smithi094 ceph-mon[40484]: pgmap v1591: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:26 smithi151 ceph-mon[32923]: pgmap v1591: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:28.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:28 smithi094 ceph-mon[40484]: pgmap v1592: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:29.018 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:28 smithi094 ceph-mon[29291]: pgmap v1592: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:29.019 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:28 smithi151 ceph-mon[32923]: pgmap v1592: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:29.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:44:29 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:44:30.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:30 smithi094 ceph-mon[29291]: pgmap v1593: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:30.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:44:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:44:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:44:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:30 smithi094 ceph-mon[40484]: pgmap v1593: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:30 smithi151 ceph-mon[32923]: pgmap v1593: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:32.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:32 smithi094 ceph-mon[29291]: pgmap v1594: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:32.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:32 smithi094 ceph-mon[40484]: pgmap v1594: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:32 smithi151 ceph-mon[32923]: pgmap v1594: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:34.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:34 smithi094 ceph-mon[40484]: pgmap v1595: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:34 smithi094 ceph-mon[29291]: pgmap v1595: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:34 smithi151 ceph-mon[32923]: pgmap v1595: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:35.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:35.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:36.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:36 smithi094 ceph-mon[29291]: pgmap v1596: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:36 smithi094 ceph-mon[40484]: pgmap v1596: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:36 smithi151 ceph-mon[32923]: pgmap v1596: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:38 smithi094 ceph-mon[29291]: pgmap v1597: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:38.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:38 smithi094 ceph-mon[40484]: pgmap v1597: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:38 smithi151 ceph-mon[32923]: pgmap v1597: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:39.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:44:39 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:44:39.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:39 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:44:39.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:39 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:44:39.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:39 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:44:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:44:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:44:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[29291]: pgmap v1598: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:44:40.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:44:40.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[40484]: pgmap v1598: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:44:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:44:40.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:40 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:40 smithi151 ceph-mon[32923]: pgmap v1598: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:40 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:40 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:40 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:40 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:40.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:40 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:44:40.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:40 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:44:40.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:40 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:44:42.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:42 smithi094 ceph-mon[29291]: pgmap v1599: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:42 smithi094 ceph-mon[40484]: pgmap v1599: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:42 smithi151 ceph-mon[32923]: pgmap v1599: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:44.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:44 smithi094 ceph-mon[29291]: pgmap v1600: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:44 smithi094 ceph-mon[40484]: pgmap v1600: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:44 smithi151 ceph-mon[32923]: pgmap v1600: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:46.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:46 smithi094 ceph-mon[29291]: pgmap v1601: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:46 smithi094 ceph-mon[40484]: pgmap v1601: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:46 smithi151 ceph-mon[32923]: pgmap v1601: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:48.767 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:48 smithi151 ceph-mon[32923]: pgmap v1602: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:48.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:48 smithi094 ceph-mon[29291]: pgmap v1602: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:48.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:48 smithi094 ceph-mon[40484]: pgmap v1602: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:49.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:44:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:44:50.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:44:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:44:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:44:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:50 smithi094 ceph-mon[29291]: pgmap v1603: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:50.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:50.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:50 smithi094 ceph-mon[40484]: pgmap v1603: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:50.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:50.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:50 smithi151 ceph-mon[32923]: pgmap v1603: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:50.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:52.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:52 smithi094 ceph-mon[29291]: pgmap v1604: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:52 smithi094 ceph-mon[40484]: pgmap v1604: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:52.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:52 smithi151 ceph-mon[32923]: pgmap v1604: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:54.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:54 smithi094 ceph-mon[29291]: pgmap v1605: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:54.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:54 smithi094 ceph-mon[40484]: pgmap v1605: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:54 smithi151 ceph-mon[32923]: pgmap v1605: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:56.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:56 smithi094 ceph-mon[29291]: pgmap v1606: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:56.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:56 smithi094 ceph-mon[40484]: pgmap v1606: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:56.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:56 smithi151 ceph-mon[32923]: pgmap v1606: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:44:58.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:44:58 smithi094 ceph-mon[29291]: pgmap v1607: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:44:58 smithi094 ceph-mon[40484]: pgmap v1607: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:44:58 smithi151 ceph-mon[32923]: pgmap v1607: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:59.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:44:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:45:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:00 smithi094 ceph-mon[29291]: pgmap v1608: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:00 smithi094 ceph-mon[40484]: pgmap v1608: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:00.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:45:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:45:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:45:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:00 smithi151 ceph-mon[32923]: pgmap v1608: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:02.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:02 smithi094 ceph-mon[40484]: pgmap v1609: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:02.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:02 smithi094 ceph-mon[29291]: pgmap v1609: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:02 smithi151 ceph-mon[32923]: pgmap v1609: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:04 smithi094 ceph-mon[29291]: pgmap v1610: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:04 smithi094 ceph-mon[40484]: pgmap v1610: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:04 smithi151 ceph-mon[32923]: pgmap v1610: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:05.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:06 smithi094 ceph-mon[29291]: pgmap v1611: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:06 smithi094 ceph-mon[40484]: pgmap v1611: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:06 smithi151 ceph-mon[32923]: pgmap v1611: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:08 smithi094 ceph-mon[29291]: pgmap v1612: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:08 smithi094 ceph-mon[40484]: pgmap v1612: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:08 smithi151 ceph-mon[32923]: pgmap v1612: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:09.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:45:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:45:10.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:10 smithi094 ceph-mon[29291]: pgmap v1613: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:10.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:10 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:45:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:45:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:45:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:10 smithi094 ceph-mon[40484]: pgmap v1613: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:10 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:10 smithi151 ceph-mon[32923]: pgmap v1613: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:10 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:12.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:12 smithi094 ceph-mon[29291]: pgmap v1614: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:12 smithi094 ceph-mon[40484]: pgmap v1614: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:12 smithi151 ceph-mon[32923]: pgmap v1614: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:14.464 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:45:14 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a[63527]: logger=cleanup t=2024-03-12T03:45:14.0211269Z level=info msg="Completed cleanup jobs" duration=7.99229ms 2024-03-12T03:45:14.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:14 smithi094 ceph-mon[29291]: pgmap v1615: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:14 smithi094 ceph-mon[40484]: pgmap v1615: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:14 smithi151 ceph-mon[32923]: pgmap v1615: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:16.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:16 smithi094 ceph-mon[29291]: pgmap v1616: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:16.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:16 smithi094 ceph-mon[40484]: pgmap v1616: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:16 smithi151 ceph-mon[32923]: pgmap v1616: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:18.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:18 smithi094 ceph-mon[29291]: pgmap v1617: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:18 smithi094 ceph-mon[40484]: pgmap v1617: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:18 smithi151 ceph-mon[32923]: pgmap v1617: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:19.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:45:19 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:45:20.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:20 smithi094 ceph-mon[40484]: pgmap v1618: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:20 smithi094 ceph-mon[29291]: pgmap v1618: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:20.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:45:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:45:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:45:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:20 smithi151 ceph-mon[32923]: pgmap v1618: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:22.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:22 smithi094 ceph-mon[29291]: pgmap v1619: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:22.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:22 smithi094 ceph-mon[40484]: pgmap v1619: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:22 smithi151 ceph-mon[32923]: pgmap v1619: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:24.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:24 smithi094 ceph-mon[29291]: pgmap v1620: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:24.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:24 smithi094 ceph-mon[40484]: pgmap v1620: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:24 smithi151 ceph-mon[32923]: pgmap v1620: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:26.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:26 smithi094 ceph-mon[29291]: pgmap v1621: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:26 smithi094 ceph-mon[40484]: pgmap v1621: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:26 smithi151 ceph-mon[32923]: pgmap v1621: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:28 smithi094 ceph-mon[29291]: pgmap v1622: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:28 smithi094 ceph-mon[40484]: pgmap v1622: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:28 smithi151 ceph-mon[32923]: pgmap v1622: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:29.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:45:29 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:45:30.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:30 smithi094 ceph-mon[29291]: pgmap v1623: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:30 smithi094 ceph-mon[40484]: pgmap v1623: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:30.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:45:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:45:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:45:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:30 smithi151 ceph-mon[32923]: pgmap v1623: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:32.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:32 smithi094 ceph-mon[40484]: pgmap v1624: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:32.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:32 smithi094 ceph-mon[29291]: pgmap v1624: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:32 smithi151 ceph-mon[32923]: pgmap v1624: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:34.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:34 smithi094 ceph-mon[40484]: pgmap v1625: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:34.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:34 smithi094 ceph-mon[29291]: pgmap v1625: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:34.966 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:34 smithi151 ceph-mon[32923]: pgmap v1625: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:35.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:35.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:36.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:36 smithi094 ceph-mon[29291]: pgmap v1626: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:36.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:36 smithi094 ceph-mon[40484]: pgmap v1626: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:36 smithi151 ceph-mon[32923]: pgmap v1626: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:38 smithi094 ceph-mon[29291]: pgmap v1627: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:38 smithi094 ceph-mon[40484]: pgmap v1627: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:38 smithi151 ceph-mon[32923]: pgmap v1627: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:39.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:45:39 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:45:40.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:45:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:45:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:45:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:40 smithi094 ceph-mon[29291]: pgmap v1628: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:40 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:45:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:40 smithi094 ceph-mon[40484]: pgmap v1628: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:40 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:45:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:40 smithi151 ceph-mon[32923]: pgmap v1628: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:40 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:45:41.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:41 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:41.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:41 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:45:41.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:41 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:45:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:41 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:41 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:45:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:41 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:45:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:41 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:41 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:45:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:41 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:45:42.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:42 smithi094 ceph-mon[40484]: pgmap v1629: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:42.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:42 smithi094 ceph-mon[29291]: pgmap v1629: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:42 smithi151 ceph-mon[32923]: pgmap v1629: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:44.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:44 smithi094 ceph-mon[40484]: pgmap v1630: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:44 smithi094 ceph-mon[29291]: pgmap v1630: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:44 smithi151 ceph-mon[32923]: pgmap v1630: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:46.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:46 smithi094 ceph-mon[29291]: pgmap v1631: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:46.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:46 smithi094 ceph-mon[40484]: pgmap v1631: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:46 smithi151 ceph-mon[32923]: pgmap v1631: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:48.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:48 smithi094 ceph-mon[40484]: pgmap v1632: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:48 smithi094 ceph-mon[29291]: pgmap v1632: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:48 smithi151 ceph-mon[32923]: pgmap v1632: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:49.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:45:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:45:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:50 smithi094 ceph-mon[29291]: pgmap v1633: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:50.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:45:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:45:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:45:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:50 smithi094 ceph-mon[40484]: pgmap v1633: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:50 smithi151 ceph-mon[32923]: pgmap v1633: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:52.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:52 smithi094 ceph-mon[29291]: pgmap v1634: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:52 smithi094 ceph-mon[40484]: pgmap v1634: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:52.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:52 smithi151 ceph-mon[32923]: pgmap v1634: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:54.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:54 smithi094 ceph-mon[29291]: pgmap v1635: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:54.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:54 smithi094 ceph-mon[40484]: pgmap v1635: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:54 smithi151 ceph-mon[32923]: pgmap v1635: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:56.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:56 smithi094 ceph-mon[29291]: pgmap v1636: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:56.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:56 smithi094 ceph-mon[40484]: pgmap v1636: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:56.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:56 smithi151 ceph-mon[32923]: pgmap v1636: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:45:58.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:45:58 smithi094 ceph-mon[29291]: pgmap v1637: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:45:58 smithi094 ceph-mon[40484]: pgmap v1637: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:45:58 smithi151 ceph-mon[32923]: pgmap v1637: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:59.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:45:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:46:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:00 smithi094 ceph-mon[29291]: pgmap v1638: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:00.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:46:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:46:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:46:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:00 smithi094 ceph-mon[40484]: pgmap v1638: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:00 smithi151 ceph-mon[32923]: pgmap v1638: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:02.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:02 smithi094 ceph-mon[29291]: pgmap v1639: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:46:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:02 smithi094 ceph-mon[40484]: pgmap v1639: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:46:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:02 smithi151 ceph-mon[32923]: pgmap v1639: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:46:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:04 smithi094 ceph-mon[29291]: pgmap v1640: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:04 smithi094 ceph-mon[40484]: pgmap v1640: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:04 smithi151 ceph-mon[32923]: pgmap v1640: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:05.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:05.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:06 smithi094 ceph-mon[29291]: pgmap v1641: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:46:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:06 smithi094 ceph-mon[40484]: pgmap v1641: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:46:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:06 smithi151 ceph-mon[32923]: pgmap v1641: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:46:08.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:08 smithi094 ceph-mon[40484]: pgmap v1642: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:08 smithi094 ceph-mon[29291]: pgmap v1642: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:08 smithi151 ceph-mon[32923]: pgmap v1642: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:09.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:46:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:46:10.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:10 smithi094 ceph-mon[29291]: pgmap v1643: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:10 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:10 smithi094 ceph-mon[40484]: pgmap v1643: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:10 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:46:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:46:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:46:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:10 smithi151 ceph-mon[32923]: pgmap v1643: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:10 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:12.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:12 smithi094 ceph-mon[29291]: pgmap v1644: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:12 smithi094 ceph-mon[40484]: pgmap v1644: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:12 smithi151 ceph-mon[32923]: pgmap v1644: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:14.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:14 smithi094 ceph-mon[29291]: pgmap v1645: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:14 smithi094 ceph-mon[40484]: pgmap v1645: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:14 smithi151 ceph-mon[32923]: pgmap v1645: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:16.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:16 smithi094 ceph-mon[29291]: pgmap v1646: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:16 smithi094 ceph-mon[40484]: pgmap v1646: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:16 smithi151 ceph-mon[32923]: pgmap v1646: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:18.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:18 smithi094 ceph-mon[29291]: pgmap v1647: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:18 smithi094 ceph-mon[40484]: pgmap v1647: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:18 smithi151 ceph-mon[32923]: pgmap v1647: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:19.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:46:19 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:46:20.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:46:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:46:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:46:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:20 smithi094 ceph-mon[40484]: pgmap v1648: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:20 smithi094 ceph-mon[29291]: pgmap v1648: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:20 smithi151 ceph-mon[32923]: pgmap v1648: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:22.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:22 smithi094 ceph-mon[29291]: pgmap v1649: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:22 smithi094 ceph-mon[40484]: pgmap v1649: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:22 smithi151 ceph-mon[32923]: pgmap v1649: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:24.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:24 smithi094 ceph-mon[29291]: pgmap v1650: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:24.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:24 smithi094 ceph-mon[40484]: pgmap v1650: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:24 smithi151 ceph-mon[32923]: pgmap v1650: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:26.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:26 smithi094 ceph-mon[29291]: pgmap v1651: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:26.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:26 smithi094 ceph-mon[29291]: osdmap e757: 8 total, 8 up, 8 in 2024-03-12T03:46:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:26 smithi094 ceph-mon[40484]: pgmap v1651: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:26 smithi094 ceph-mon[40484]: osdmap e757: 8 total, 8 up, 8 in 2024-03-12T03:46:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:26 smithi151 ceph-mon[32923]: pgmap v1651: 228 pgs: 228 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:26 smithi151 ceph-mon[32923]: osdmap e757: 8 total, 8 up, 8 in 2024-03-12T03:46:27.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:27 smithi094 ceph-mon[29291]: osdmap e758: 8 total, 8 up, 8 in 2024-03-12T03:46:27.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:27 smithi094 ceph-mon[40484]: osdmap e758: 8 total, 8 up, 8 in 2024-03-12T03:46:27.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:27 smithi151 ceph-mon[32923]: osdmap e758: 8 total, 8 up, 8 in 2024-03-12T03:46:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:28 smithi094 ceph-mon[29291]: pgmap v1654: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:46:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:28 smithi094 ceph-mon[29291]: osdmap e759: 8 total, 8 up, 8 in 2024-03-12T03:46:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:28 smithi094 ceph-mon[40484]: pgmap v1654: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:46:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:28 smithi094 ceph-mon[40484]: osdmap e759: 8 total, 8 up, 8 in 2024-03-12T03:46:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:28 smithi151 ceph-mon[32923]: pgmap v1654: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:46:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:28 smithi151 ceph-mon[32923]: osdmap e759: 8 total, 8 up, 8 in 2024-03-12T03:46:29.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:46:29 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:46:30.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:29 smithi094 ceph-mon[29291]: osdmap e760: 8 total, 8 up, 8 in 2024-03-12T03:46:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:29 smithi094 ceph-mon[40484]: osdmap e760: 8 total, 8 up, 8 in 2024-03-12T03:46:30.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:29 smithi151 ceph-mon[32923]: osdmap e760: 8 total, 8 up, 8 in 2024-03-12T03:46:30.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:30 smithi094 ceph-mon[40484]: pgmap v1657: 164 pgs: 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:30.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:30.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:30 smithi094 ceph-mon[40484]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:30.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:30 smithi094 ceph-mon[40484]: osdmap e761: 8 total, 8 up, 8 in 2024-03-12T03:46:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:30 smithi094 ceph-mon[29291]: pgmap v1657: 164 pgs: 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:30 smithi094 ceph-mon[29291]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:30 smithi094 ceph-mon[29291]: osdmap e761: 8 total, 8 up, 8 in 2024-03-12T03:46:30.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:46:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:46:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:46:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:30 smithi151 ceph-mon[32923]: pgmap v1657: 164 pgs: 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:30 smithi151 ceph-mon[32923]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:30 smithi151 ceph-mon[32923]: osdmap e761: 8 total, 8 up, 8 in 2024-03-12T03:46:31.740 INFO:tasks.workunit.client.0.smithi094.stdout: list: Running main() from gmock_main.cc 2024-03-12T03:46:31.740 INFO:tasks.workunit.client.0.smithi094.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-03-12T03:46:31.740 INFO:tasks.workunit.client.0.smithi094.stdout: list: [----------] Global test environment set-up. 2024-03-12T03:46:31.741 INFO:tasks.workunit.client.0.smithi094.stdout: list: [----------] 3 tests from NeoradosList 2024-03-12T03:46:31.741 INFO:tasks.workunit.client.0.smithi094.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-03-12T03:46:31.741 INFO:tasks.workunit.client.0.smithi094.stdout: list: [ OK ] NeoradosList.ListObjects (1808617 ms) 2024-03-12T03:46:31.741 INFO:tasks.workunit.client.0.smithi094.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-03-12T03:46:31.741 INFO:tasks.workunit.client.0.smithi094.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3015 ms) 2024-03-12T03:46:31.741 INFO:tasks.workunit.client.0.smithi094.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-03-12T03:46:31.741 INFO:tasks.workunit.client.0.smithi094.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3037 ms) 2024-03-12T03:46:31.741 INFO:tasks.workunit.client.0.smithi094.stdout: list: [----------] 3 tests from NeoradosList (1814669 ms total) 2024-03-12T03:46:31.741 INFO:tasks.workunit.client.0.smithi094.stdout: list: 2024-03-12T03:46:31.741 INFO:tasks.workunit.client.0.smithi094.stdout: list: [----------] Global test environment tear-down 2024-03-12T03:46:31.741 INFO:tasks.workunit.client.0.smithi094.stdout: list: [==========] 3 tests from 1 test suite ran. (1814669 ms total) 2024-03-12T03:46:31.741 INFO:tasks.workunit.client.0.smithi094.stdout: list: [ PASSED ] 3 tests. 2024-03-12T03:46:32.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:31 smithi094 ceph-mon[29291]: osdmap e762: 8 total, 8 up, 8 in 2024-03-12T03:46:32.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:31 smithi094 ceph-mon[40484]: osdmap e762: 8 total, 8 up, 8 in 2024-03-12T03:46:32.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:31 smithi151 ceph-mon[32923]: osdmap e762: 8 total, 8 up, 8 in 2024-03-12T03:46:33.030 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:32 smithi094 ceph-mon[29291]: pgmap v1660: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:33.030 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:32 smithi094 ceph-mon[29291]: osdmap e763: 8 total, 8 up, 8 in 2024-03-12T03:46:33.031 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:32 smithi094 ceph-mon[40484]: pgmap v1660: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:33.031 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:32 smithi094 ceph-mon[40484]: osdmap e763: 8 total, 8 up, 8 in 2024-03-12T03:46:33.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:32 smithi151 ceph-mon[32923]: pgmap v1660: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:33.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:32 smithi151 ceph-mon[32923]: osdmap e763: 8 total, 8 up, 8 in 2024-03-12T03:46:34.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:33 smithi094 ceph-mon[29291]: osdmap e764: 8 total, 8 up, 8 in 2024-03-12T03:46:34.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:33 smithi094 ceph-mon[40484]: osdmap e764: 8 total, 8 up, 8 in 2024-03-12T03:46:34.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:33 smithi151 ceph-mon[32923]: osdmap e764: 8 total, 8 up, 8 in 2024-03-12T03:46:35.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:34 smithi094 ceph-mon[29291]: pgmap v1663: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:35.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:34 smithi094 ceph-mon[29291]: osdmap e765: 8 total, 8 up, 8 in 2024-03-12T03:46:35.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:34 smithi094 ceph-mon[40484]: pgmap v1663: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:35.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:34 smithi094 ceph-mon[40484]: osdmap e765: 8 total, 8 up, 8 in 2024-03-12T03:46:35.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:34 smithi151 ceph-mon[32923]: pgmap v1663: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:35.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:34 smithi151 ceph-mon[32923]: osdmap e765: 8 total, 8 up, 8 in 2024-03-12T03:46:36.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:35 smithi094 ceph-mon[40484]: osdmap e766: 8 total, 8 up, 8 in 2024-03-12T03:46:36.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:36.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:35 smithi094 ceph-mon[29291]: osdmap e766: 8 total, 8 up, 8 in 2024-03-12T03:46:36.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:35 smithi151 ceph-mon[32923]: osdmap e766: 8 total, 8 up, 8 in 2024-03-12T03:46:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:37.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:36 smithi094 ceph-mon[29291]: pgmap v1666: 164 pgs: 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:37.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:36 smithi094 ceph-mon[29291]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:37.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:36 smithi094 ceph-mon[29291]: osdmap e767: 8 total, 8 up, 8 in 2024-03-12T03:46:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:36 smithi094 ceph-mon[40484]: pgmap v1666: 164 pgs: 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:36 smithi094 ceph-mon[40484]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:36 smithi094 ceph-mon[40484]: osdmap e767: 8 total, 8 up, 8 in 2024-03-12T03:46:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:36 smithi151 ceph-mon[32923]: pgmap v1666: 164 pgs: 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:36 smithi151 ceph-mon[32923]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:36 smithi151 ceph-mon[32923]: osdmap e767: 8 total, 8 up, 8 in 2024-03-12T03:46:38.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:37 smithi094 ceph-mon[29291]: osdmap e768: 8 total, 8 up, 8 in 2024-03-12T03:46:38.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:37 smithi094 ceph-mon[40484]: osdmap e768: 8 total, 8 up, 8 in 2024-03-12T03:46:38.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:37 smithi151 ceph-mon[32923]: osdmap e768: 8 total, 8 up, 8 in 2024-03-12T03:46:39.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:38 smithi094 ceph-mon[29291]: pgmap v1669: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:46:39.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:38 smithi094 ceph-mon[29291]: osdmap e769: 8 total, 8 up, 8 in 2024-03-12T03:46:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:38 smithi094 ceph-mon[40484]: pgmap v1669: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:46:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:38 smithi094 ceph-mon[40484]: osdmap e769: 8 total, 8 up, 8 in 2024-03-12T03:46:39.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:38 smithi151 ceph-mon[32923]: pgmap v1669: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:46:39.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:38 smithi151 ceph-mon[32923]: osdmap e769: 8 total, 8 up, 8 in 2024-03-12T03:46:39.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:46:39 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:46:40.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:39 smithi094 ceph-mon[29291]: osdmap e770: 8 total, 8 up, 8 in 2024-03-12T03:46:40.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:39 smithi094 ceph-mon[40484]: osdmap e770: 8 total, 8 up, 8 in 2024-03-12T03:46:40.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:39 smithi151 ceph-mon[32923]: osdmap e770: 8 total, 8 up, 8 in 2024-03-12T03:46:40.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:40 smithi094 ceph-mon[29291]: pgmap v1672: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:40.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:40.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:40 smithi094 ceph-mon[29291]: osdmap e771: 8 total, 8 up, 8 in 2024-03-12T03:46:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:40 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:46:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:40 smithi094 ceph-mon[40484]: pgmap v1672: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:40 smithi094 ceph-mon[40484]: osdmap e771: 8 total, 8 up, 8 in 2024-03-12T03:46:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:40 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:46:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:46:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:46:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:46:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:40 smithi151 ceph-mon[32923]: pgmap v1672: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:40 smithi151 ceph-mon[32923]: osdmap e771: 8 total, 8 up, 8 in 2024-03-12T03:46:41.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:40 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:46:42.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:41 smithi094 ceph-mon[29291]: osdmap e772: 8 total, 8 up, 8 in 2024-03-12T03:46:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:41 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:46:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:41 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:46:42.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:41 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:46:42.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:41 smithi094 ceph-mon[40484]: osdmap e772: 8 total, 8 up, 8 in 2024-03-12T03:46:42.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:41 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:46:42.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:41 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:46:42.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:41 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:46:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:41 smithi151 ceph-mon[32923]: osdmap e772: 8 total, 8 up, 8 in 2024-03-12T03:46:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:41 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:46:42.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:41 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:46:42.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:41 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:46:43.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:42 smithi094 ceph-mon[29291]: pgmap v1675: 164 pgs: 164 active+clean; 582 KiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:43.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:42 smithi094 ceph-mon[29291]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:42 smithi094 ceph-mon[29291]: osdmap e773: 8 total, 8 up, 8 in 2024-03-12T03:46:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:42 smithi094 ceph-mon[40484]: pgmap v1675: 164 pgs: 164 active+clean; 582 KiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:42 smithi094 ceph-mon[40484]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:42 smithi094 ceph-mon[40484]: osdmap e773: 8 total, 8 up, 8 in 2024-03-12T03:46:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:42 smithi151 ceph-mon[32923]: pgmap v1675: 164 pgs: 164 active+clean; 582 KiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:42 smithi151 ceph-mon[32923]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:43.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:42 smithi151 ceph-mon[32923]: osdmap e773: 8 total, 8 up, 8 in 2024-03-12T03:46:44.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:43 smithi094 ceph-mon[29291]: osdmap e774: 8 total, 8 up, 8 in 2024-03-12T03:46:44.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:43 smithi094 ceph-mon[40484]: osdmap e774: 8 total, 8 up, 8 in 2024-03-12T03:46:44.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:43 smithi151 ceph-mon[32923]: osdmap e774: 8 total, 8 up, 8 in 2024-03-12T03:46:45.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:44 smithi094 ceph-mon[29291]: pgmap v1678: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:45.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:44 smithi094 ceph-mon[29291]: osdmap e775: 8 total, 8 up, 8 in 2024-03-12T03:46:45.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:44 smithi094 ceph-mon[40484]: pgmap v1678: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:45.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:44 smithi094 ceph-mon[40484]: osdmap e775: 8 total, 8 up, 8 in 2024-03-12T03:46:45.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:44 smithi151 ceph-mon[32923]: pgmap v1678: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:45.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:44 smithi151 ceph-mon[32923]: osdmap e775: 8 total, 8 up, 8 in 2024-03-12T03:46:46.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:45 smithi094 ceph-mon[29291]: osdmap e776: 8 total, 8 up, 8 in 2024-03-12T03:46:46.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:45 smithi094 ceph-mon[40484]: osdmap e776: 8 total, 8 up, 8 in 2024-03-12T03:46:46.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:45 smithi151 ceph-mon[32923]: osdmap e776: 8 total, 8 up, 8 in 2024-03-12T03:46:47.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:46 smithi094 ceph-mon[29291]: pgmap v1681: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:47.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:46 smithi094 ceph-mon[29291]: osdmap e777: 8 total, 8 up, 8 in 2024-03-12T03:46:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:46 smithi094 ceph-mon[40484]: pgmap v1681: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:46 smithi094 ceph-mon[40484]: osdmap e777: 8 total, 8 up, 8 in 2024-03-12T03:46:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:46 smithi151 ceph-mon[32923]: pgmap v1681: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:46 smithi151 ceph-mon[32923]: osdmap e777: 8 total, 8 up, 8 in 2024-03-12T03:46:48.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:47 smithi151 ceph-mon[32923]: osdmap e778: 8 total, 8 up, 8 in 2024-03-12T03:46:48.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:47 smithi094 ceph-mon[29291]: osdmap e778: 8 total, 8 up, 8 in 2024-03-12T03:46:48.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:47 smithi094 ceph-mon[40484]: osdmap e778: 8 total, 8 up, 8 in 2024-03-12T03:46:49.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:48 smithi094 ceph-mon[29291]: pgmap v1684: 164 pgs: 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:46:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:48 smithi094 ceph-mon[29291]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:48 smithi094 ceph-mon[29291]: osdmap e779: 8 total, 8 up, 8 in 2024-03-12T03:46:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:48 smithi094 ceph-mon[40484]: pgmap v1684: 164 pgs: 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:46:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:48 smithi094 ceph-mon[40484]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:48 smithi094 ceph-mon[40484]: osdmap e779: 8 total, 8 up, 8 in 2024-03-12T03:46:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:48 smithi151 ceph-mon[32923]: pgmap v1684: 164 pgs: 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:46:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:48 smithi151 ceph-mon[32923]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:49.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:48 smithi151 ceph-mon[32923]: osdmap e779: 8 total, 8 up, 8 in 2024-03-12T03:46:49.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:46:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:46:50.110 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:49 smithi094 ceph-mon[29291]: osdmap e780: 8 total, 8 up, 8 in 2024-03-12T03:46:50.111 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:49 smithi094 ceph-mon[40484]: osdmap e780: 8 total, 8 up, 8 in 2024-03-12T03:46:50.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:49 smithi151 ceph-mon[32923]: osdmap e780: 8 total, 8 up, 8 in 2024-03-12T03:46:50.836 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:46:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:46:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:46:51.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:50 smithi094 ceph-mon[29291]: pgmap v1687: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:51.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:51.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:50 smithi094 ceph-mon[29291]: osdmap e781: 8 total, 8 up, 8 in 2024-03-12T03:46:51.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:50 smithi094 ceph-mon[40484]: pgmap v1687: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:51.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:51.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:50 smithi094 ceph-mon[40484]: osdmap e781: 8 total, 8 up, 8 in 2024-03-12T03:46:51.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:50 smithi151 ceph-mon[32923]: pgmap v1687: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:50 smithi151 ceph-mon[32923]: osdmap e781: 8 total, 8 up, 8 in 2024-03-12T03:46:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:52.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:51 smithi094 ceph-mon[29291]: osdmap e782: 8 total, 8 up, 8 in 2024-03-12T03:46:52.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:51 smithi094 ceph-mon[29291]: pgmap v1690: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:52.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:51 smithi094 ceph-mon[40484]: osdmap e782: 8 total, 8 up, 8 in 2024-03-12T03:46:52.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:51 smithi094 ceph-mon[40484]: pgmap v1690: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:52.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:51 smithi151 ceph-mon[32923]: osdmap e782: 8 total, 8 up, 8 in 2024-03-12T03:46:52.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:51 smithi151 ceph-mon[32923]: pgmap v1690: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:53.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:52 smithi094 ceph-mon[29291]: osdmap e783: 8 total, 8 up, 8 in 2024-03-12T03:46:53.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:52 smithi094 ceph-mon[40484]: osdmap e783: 8 total, 8 up, 8 in 2024-03-12T03:46:53.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:52 smithi151 ceph-mon[32923]: osdmap e783: 8 total, 8 up, 8 in 2024-03-12T03:46:54.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:53 smithi094 ceph-mon[29291]: osdmap e784: 8 total, 8 up, 8 in 2024-03-12T03:46:54.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:53 smithi094 ceph-mon[29291]: pgmap v1693: 164 pgs: 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:54.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:53 smithi094 ceph-mon[29291]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:54.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:53 smithi094 ceph-mon[40484]: osdmap e784: 8 total, 8 up, 8 in 2024-03-12T03:46:54.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:53 smithi094 ceph-mon[40484]: pgmap v1693: 164 pgs: 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:54.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:53 smithi094 ceph-mon[40484]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:54.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:53 smithi151 ceph-mon[32923]: osdmap e784: 8 total, 8 up, 8 in 2024-03-12T03:46:54.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:53 smithi151 ceph-mon[32923]: pgmap v1693: 164 pgs: 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:54.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:53 smithi151 ceph-mon[32923]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:46:55.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:54 smithi094 ceph-mon[29291]: osdmap e785: 8 total, 8 up, 8 in 2024-03-12T03:46:55.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:54 smithi094 ceph-mon[40484]: osdmap e785: 8 total, 8 up, 8 in 2024-03-12T03:46:55.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:54 smithi151 ceph-mon[32923]: osdmap e785: 8 total, 8 up, 8 in 2024-03-12T03:46:56.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:55 smithi094 ceph-mon[29291]: osdmap e786: 8 total, 8 up, 8 in 2024-03-12T03:46:56.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:55 smithi094 ceph-mon[29291]: pgmap v1696: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:55 smithi094 ceph-mon[40484]: osdmap e786: 8 total, 8 up, 8 in 2024-03-12T03:46:56.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:55 smithi094 ceph-mon[40484]: pgmap v1696: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:56.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:55 smithi151 ceph-mon[32923]: osdmap e786: 8 total, 8 up, 8 in 2024-03-12T03:46:56.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:55 smithi151 ceph-mon[32923]: pgmap v1696: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:57.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:56 smithi094 ceph-mon[29291]: osdmap e787: 8 total, 8 up, 8 in 2024-03-12T03:46:57.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:56 smithi094 ceph-mon[40484]: osdmap e787: 8 total, 8 up, 8 in 2024-03-12T03:46:57.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:56 smithi151 ceph-mon[32923]: osdmap e787: 8 total, 8 up, 8 in 2024-03-12T03:46:58.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:57 smithi094 ceph-mon[29291]: osdmap e788: 8 total, 8 up, 8 in 2024-03-12T03:46:58.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:57 smithi094 ceph-mon[29291]: pgmap v1699: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:46:58.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:57 smithi094 ceph-mon[40484]: osdmap e788: 8 total, 8 up, 8 in 2024-03-12T03:46:58.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:57 smithi094 ceph-mon[40484]: pgmap v1699: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:46:58.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:57 smithi151 ceph-mon[32923]: osdmap e788: 8 total, 8 up, 8 in 2024-03-12T03:46:58.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:57 smithi151 ceph-mon[32923]: pgmap v1699: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:46:59.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:58 smithi094 ceph-mon[29291]: osdmap e789: 8 total, 8 up, 8 in 2024-03-12T03:46:59.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:58 smithi094 ceph-mon[40484]: osdmap e789: 8 total, 8 up, 8 in 2024-03-12T03:46:59.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:58 smithi151 ceph-mon[32923]: osdmap e789: 8 total, 8 up, 8 in 2024-03-12T03:46:59.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:46:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:47:00.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:59 smithi094 ceph-mon[29291]: osdmap e790: 8 total, 8 up, 8 in 2024-03-12T03:47:00.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:59 smithi094 ceph-mon[29291]: pgmap v1702: 164 pgs: 164 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:00.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:46:59 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:00.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:59 smithi094 ceph-mon[40484]: osdmap e790: 8 total, 8 up, 8 in 2024-03-12T03:47:00.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:59 smithi094 ceph-mon[40484]: pgmap v1702: 164 pgs: 164 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:00.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:46:59 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:00.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:59 smithi151 ceph-mon[32923]: osdmap e790: 8 total, 8 up, 8 in 2024-03-12T03:47:00.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:59 smithi151 ceph-mon[32923]: pgmap v1702: 164 pgs: 164 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:00.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:46:59 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:00.903 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:47:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:47:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:47:01.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:00 smithi094 ceph-mon[29291]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:47:01.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:00 smithi094 ceph-mon[29291]: osdmap e791: 8 total, 8 up, 8 in 2024-03-12T03:47:01.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:00 smithi094 ceph-mon[40484]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:47:01.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:00 smithi094 ceph-mon[40484]: osdmap e791: 8 total, 8 up, 8 in 2024-03-12T03:47:01.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:00 smithi151 ceph-mon[32923]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:47:01.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:00 smithi151 ceph-mon[32923]: osdmap e791: 8 total, 8 up, 8 in 2024-03-12T03:47:02.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:01 smithi094 ceph-mon[29291]: osdmap e792: 8 total, 8 up, 8 in 2024-03-12T03:47:02.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:01 smithi094 ceph-mon[29291]: pgmap v1705: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:02.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:01 smithi094 ceph-mon[40484]: osdmap e792: 8 total, 8 up, 8 in 2024-03-12T03:47:02.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:01 smithi094 ceph-mon[40484]: pgmap v1705: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:02.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:01 smithi151 ceph-mon[32923]: osdmap e792: 8 total, 8 up, 8 in 2024-03-12T03:47:02.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:01 smithi151 ceph-mon[32923]: pgmap v1705: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:03.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:02 smithi094 ceph-mon[29291]: osdmap e793: 8 total, 8 up, 8 in 2024-03-12T03:47:03.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:02 smithi094 ceph-mon[40484]: osdmap e793: 8 total, 8 up, 8 in 2024-03-12T03:47:03.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:02 smithi151 ceph-mon[32923]: osdmap e793: 8 total, 8 up, 8 in 2024-03-12T03:47:04.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:03 smithi094 ceph-mon[40484]: osdmap e794: 8 total, 8 up, 8 in 2024-03-12T03:47:04.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:03 smithi094 ceph-mon[40484]: pgmap v1708: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:04.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:03 smithi094 ceph-mon[29291]: osdmap e794: 8 total, 8 up, 8 in 2024-03-12T03:47:04.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:03 smithi094 ceph-mon[29291]: pgmap v1708: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:04.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:03 smithi151 ceph-mon[32923]: osdmap e794: 8 total, 8 up, 8 in 2024-03-12T03:47:04.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:03 smithi151 ceph-mon[32923]: pgmap v1708: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:04.944 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: Running main() from gmock_main.cc 2024-03-12T03:47:04.945 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-03-12T03:47:04.945 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [----------] Global test environment set-up. 2024-03-12T03:47:04.945 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-03-12T03:47:04.945 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-03-12T03:47:04.945 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (1808570 ms) 2024-03-12T03:47:04.945 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-03-12T03:47:04.945 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3014 ms) 2024-03-12T03:47:04.945 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-03-12T03:47:04.945 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3036 ms) 2024-03-12T03:47:04.945 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-03-12T03:47:04.945 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3019 ms) 2024-03-12T03:47:04.946 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-03-12T03:47:04.946 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3016 ms) 2024-03-12T03:47:04.946 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-03-12T03:47:04.946 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3019 ms) 2024-03-12T03:47:04.946 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-03-12T03:47:04.946 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3015 ms) 2024-03-12T03:47:04.946 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-03-12T03:47:04.946 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3012 ms) 2024-03-12T03:47:04.946 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-03-12T03:47:04.946 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3025 ms) 2024-03-12T03:47:04.946 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-03-12T03:47:04.946 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3023 ms) 2024-03-12T03:47:04.947 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-03-12T03:47:04.947 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3014 ms) 2024-03-12T03:47:04.947 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-03-12T03:47:04.947 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3023 ms) 2024-03-12T03:47:04.947 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-03-12T03:47:04.947 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3020 ms) 2024-03-12T03:47:04.947 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-03-12T03:47:04.947 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3016 ms) 2024-03-12T03:47:04.947 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (1847825 ms total) 2024-03-12T03:47:04.947 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: 2024-03-12T03:47:04.947 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [----------] Global test environment tear-down 2024-03-12T03:47:04.948 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (1847825 ms total) 2024-03-12T03:47:04.948 INFO:tasks.workunit.client.0.smithi094.stdout: read_operations: [ PASSED ] 14 tests. 2024-03-12T03:47:04.951 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.951 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78443 2024-03-12T03:47:04.951 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78443 2024-03-12T03:47:04.951 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.951 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78983 2024-03-12T03:47:04.951 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78983 2024-03-12T03:47:04.951 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.951 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79357 2024-03-12T03:47:04.951 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 79357 2024-03-12T03:47:04.951 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.951 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78344 2024-03-12T03:47:04.952 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78344 2024-03-12T03:47:04.952 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.952 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78740 2024-03-12T03:47:04.952 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78740 2024-03-12T03:47:04.952 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.952 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79119 2024-03-12T03:47:04.952 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 79119 2024-03-12T03:47:04.952 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.952 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78871 2024-03-12T03:47:04.952 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78871 2024-03-12T03:47:04.952 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.952 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79251 2024-03-12T03:47:04.953 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 79251 2024-03-12T03:47:04.953 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.953 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78655 2024-03-12T03:47:04.953 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78655 2024-03-12T03:47:04.953 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.953 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78227 2024-03-12T03:47:04.953 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78227 2024-03-12T03:47:04.953 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.953 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79327 2024-03-12T03:47:04.953 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 79327 2024-03-12T03:47:04.953 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.954 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78586 2024-03-12T03:47:04.954 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78586 2024-03-12T03:47:04.954 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.954 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78013 2024-03-12T03:47:04.954 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78013 2024-03-12T03:47:04.954 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.954 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78071 2024-03-12T03:47:04.954 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78071 2024-03-12T03:47:04.954 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.954 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78495 2024-03-12T03:47:04.954 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78495 2024-03-12T03:47:04.954 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.955 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78115 2024-03-12T03:47:04.955 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78115 2024-03-12T03:47:04.955 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.955 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78599 2024-03-12T03:47:04.955 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78599 2024-03-12T03:47:04.955 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.955 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=78619 2024-03-12T03:47:04.955 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 78619 2024-03-12T03:47:04.955 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.955 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79096 2024-03-12T03:47:04.955 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 79096 2024-03-12T03:47:04.955 INFO:tasks.workunit.client.0.smithi094.stderr:+ for t in "${!pids[@]}" 2024-03-12T03:47:04.955 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=79381 2024-03-12T03:47:04.956 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 79381 2024-03-12T03:47:04.956 INFO:tasks.workunit.client.0.smithi094.stderr:+ exit 0 2024-03-12T03:47:04.956 INFO:tasks.workunit.client.0.smithi094.stderr:+ cleanup 2024-03-12T03:47:04.956 INFO:tasks.workunit.client.0.smithi094.stderr:+ pkill -P 78007 2024-03-12T03:47:04.960 INFO:tasks.workunit.client.0.smithi094.stderr:+ true 2024-03-12T03:47:04.961 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-12T03:47:04.961 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-12T03:47:04.994 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-03-12T03:47:04.996 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-03-12T03:47:04.996 DEBUG:teuthology.orchestra.run.smithi094:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-03-12T03:47:05.055 INFO:tasks.workunit.client.0.smithi094.stderr:++ uuidgen 2024-03-12T03:47:05.062 INFO:tasks.workunit.client.0.smithi094.stderr:+ p=f98a828b-0804-4d73-9e78-2d8570d1ebe3 2024-03-12T03:47:05.062 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool create f98a828b-0804-4d73-9e78-2d8570d1ebe3 12 2024-03-12T03:47:05.149 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.144+0000 7fce1bfff640 1 -- 172.21.15.94:0/2370700631 >> v1:172.21.15.94:6789/0 conn(0x7fce1c109e60 legacy=0x7fce1c10c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:05.149 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.144+0000 7fce1bfff640 1 -- 172.21.15.94:0/2370700631 shutdown_connections 2024-03-12T03:47:05.150 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.144+0000 7fce1bfff640 1 -- 172.21.15.94:0/2370700631 >> 172.21.15.94:0/2370700631 conn(0x7fce1c0fca60 msgr2=0x7fce1c0fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:47:05.150 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.144+0000 7fce1bfff640 1 -- 172.21.15.94:0/2370700631 shutdown_connections 2024-03-12T03:47:05.150 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.144+0000 7fce1bfff640 1 -- 172.21.15.94:0/2370700631 wait complete. 2024-03-12T03:47:05.150 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.144+0000 7fce1bfff640 1 Processor -- start 2024-03-12T03:47:05.150 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce1bfff640 1 -- start start 2024-03-12T03:47:05.150 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce1bfff640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce1c1a5fb0 con 0x7fce1c10d9d0 2024-03-12T03:47:05.150 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce1bfff640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce1c1a71f0 con 0x7fce1c1011e0 2024-03-12T03:47:05.150 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce1bfff640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce1c1a8430 con 0x7fce1c109e60 2024-03-12T03:47:05.150 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce1a7fc640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fce1c109e60 0x7fce1c1a0e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:60890/0 (socket says 172.21.15.94:60890) 2024-03-12T03:47:05.150 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce1a7fc640 1 -- 172.21.15.94:0/3299069758 learned_addr learned my addr 172.21.15.94:0/3299069758 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:47:05.150 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055433879 0 0) 0x7fce1c1a5fb0 con 0x7fce1c10d9d0 2024-03-12T03:47:05.151 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce04003240 con 0x7fce1c10d9d0 2024-03-12T03:47:05.151 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810207472 0 0) 0x7fce1c1a8430 con 0x7fce1c109e60 2024-03-12T03:47:05.151 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce1c1a5fb0 con 0x7fce1c109e60 2024-03-12T03:47:05.151 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 556478451 0 0) 0x7fce04003240 con 0x7fce1c10d9d0 2024-03-12T03:47:05.151 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce1c1a8430 con 0x7fce1c10d9d0 2024-03-12T03:47:05.151 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fce0c0017f0 con 0x7fce1c10d9d0 2024-03-12T03:47:05.151 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1807198712 0 0) 0x7fce1c1a5fb0 con 0x7fce1c109e60 2024-03-12T03:47:05.151 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce04003240 con 0x7fce1c109e60 2024-03-12T03:47:05.151 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fce10004800 con 0x7fce1c109e60 2024-03-12T03:47:05.151 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326884595 0 0) 0x7fce1c1a71f0 con 0x7fce1c1011e0 2024-03-12T03:47:05.151 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce1c1a5fb0 con 0x7fce1c1011e0 2024-03-12T03:47:05.151 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 858657380 0 0) 0x7fce1c1a8430 con 0x7fce1c10d9d0 2024-03-12T03:47:05.151 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.145+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 >> v1:172.21.15.94:6790/0 conn(0x7fce1c109e60 legacy=0x7fce1c1a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:05.152 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.146+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 >> v1:172.21.15.151:6789/0 conn(0x7fce1c1011e0 legacy=0x7fce1c1064d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:05.152 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.146+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1c1a9670 con 0x7fce1c10d9d0 2024-03-12T03:47:05.152 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.146+0000 7fce1bfff640 1 -- 172.21.15.94:0/3299069758 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1c1a8660 con 0x7fce1c10d9d0 2024-03-12T03:47:05.152 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.146+0000 7fce1bfff640 1 -- 172.21.15.94:0/3299069758 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1c1a8b30 con 0x7fce1c10d9d0 2024-03-12T03:47:05.152 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.146+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fce0c003a80 con 0x7fce1c10d9d0 2024-03-12T03:47:05.152 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.146+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fce0c004980 con 0x7fce1c10d9d0 2024-03-12T03:47:05.152 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.146+0000 7fce1bfff640 1 -- 172.21.15.94:0/3299069758 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce08005080 con 0x7fce1c10d9d0 2024-03-12T03:47:05.152 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.147+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7fce0c0035f0 con 0x7fce1c10d9d0 2024-03-12T03:47:05.153 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.147+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(796..796 src has 254..796) v4 ==== 7442+0+0 (unknown 2983440712 0 0) 0x7fce0c0884b0 con 0x7fce1c10d9d0 2024-03-12T03:47:05.154 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.152+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fce0c055850 con 0x7fce1c10d9d0 2024-03-12T03:47:05.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:04 smithi094 ceph-mon[29291]: osdmap e795: 8 total, 8 up, 8 in 2024-03-12T03:47:05.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:04 smithi094 ceph-mon[40484]: osdmap e795: 8 total, 8 up, 8 in 2024-03-12T03:47:05.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:04 smithi151 ceph-mon[32923]: osdmap e795: 8 total, 8 up, 8 in 2024-03-12T03:47:05.336 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.334+0000 7fce1bfff640 1 -- 172.21.15.94:0/3299069758 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pg_num": 12} v 0) v1 -- 0x7fce08005370 con 0x7fce1c10d9d0 2024-03-12T03:47:05.939 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:05.936+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pg_num": 12}]=0 pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' created v797) v1 ==== 176+0+0 (unknown 2401911999 0 0) 0x7fce0c0592a0 con 0x7fce1c10d9d0 2024-03-12T03:47:06.049 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.046+0000 7fce1bfff640 1 -- 172.21.15.94:0/3299069758 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pg_num": 12} v 0) v1 -- 0x7fce08001170 con 0x7fce1c10d9d0 2024-03-12T03:47:06.049 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.046+0000 7fce03fff640 1 -- 172.21.15.94:0/3299069758 <== mon.0 v1:172.21.15.94:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pg_num": 12}]=0 pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' already exists v797) v1 ==== 183+0+0 (unknown 1593313894 0 0) 0x7fce08001170 con 0x7fce1c10d9d0 2024-03-12T03:47:06.049 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' already exists 2024-03-12T03:47:06.051 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.048+0000 7fce1bfff640 1 -- 172.21.15.94:0/3299069758 >> v1:172.21.15.94:6800/2099808905 conn(0x7fce0406ece0 legacy=0x7fce040711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:06.051 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.048+0000 7fce1bfff640 1 -- 172.21.15.94:0/3299069758 >> v1:172.21.15.94:6789/0 conn(0x7fce1c10d9d0 legacy=0x7fce1c1a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:06.051 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.048+0000 7fce1bfff640 1 -- 172.21.15.94:0/3299069758 shutdown_connections 2024-03-12T03:47:06.051 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.048+0000 7fce1bfff640 1 -- 172.21.15.94:0/3299069758 >> 172.21.15.94:0/3299069758 conn(0x7fce1c0fca60 msgr2=0x7fce1c110ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:47:06.051 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.048+0000 7fce1bfff640 1 -- 172.21.15.94:0/3299069758 shutdown_connections 2024-03-12T03:47:06.051 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.048+0000 7fce1bfff640 1 -- 172.21.15.94:0/3299069758 wait complete. 2024-03-12T03:47:06.060 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool set-quota f98a828b-0804-4d73-9e78-2d8570d1ebe3 max_objects 10 2024-03-12T03:47:06.144 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.139+0000 7f5c928ea640 1 -- 172.21.15.94:0/2737875702 >> v1:172.21.15.94:6790/0 conn(0x7f5c8c0ffbd0 legacy=0x7f5c8c0fffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:06.144 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.139+0000 7f5c928ea640 1 -- 172.21.15.94:0/2737875702 shutdown_connections 2024-03-12T03:47:06.144 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.139+0000 7f5c928ea640 1 -- 172.21.15.94:0/2737875702 >> 172.21.15.94:0/2737875702 conn(0x7f5c8c075350 msgr2=0x7f5c8c075760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.139+0000 7f5c928ea640 1 -- 172.21.15.94:0/2737875702 shutdown_connections 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.139+0000 7f5c928ea640 1 -- 172.21.15.94:0/2737875702 wait complete. 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.140+0000 7f5c928ea640 1 Processor -- start 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.140+0000 7f5c928ea640 1 -- start start 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.140+0000 7f5c928ea640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c8c1a5f70 con 0x7f5c8c0ffbd0 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.140+0000 7f5c928ea640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c8c1a7190 con 0x7f5c8c10dbb0 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.140+0000 7f5c928ea640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c8c1a83d0 con 0x7f5c8c10a040 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.140+0000 7f5c8bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5c8c0ffbd0 0x7f5c8c106510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43686/0 (socket says 172.21.15.94:43686) 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.140+0000 7f5c8bfff640 1 -- 172.21.15.94:0/1446764348 learned_addr learned my addr 172.21.15.94:0/1446764348 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.140+0000 7f5c8b7fe640 1 --1- 172.21.15.94:0/1446764348 >> v1:172.21.15.94:6790/0 conn(0x7f5c8c10a040 0x7f5c8c1a1000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:60902/0 (socket says 172.21.15.94:60902) 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402688546 0 0) 0x7f5c8c1a83d0 con 0x7f5c8c10a040 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c74003520 con 0x7f5c8c10a040 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649202565 0 0) 0x7f5c8c1a5f70 con 0x7f5c8c0ffbd0 2024-03-12T03:47:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8c1a83d0 con 0x7f5c8c0ffbd0 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723448356 0 0) 0x7f5c8c1a7190 con 0x7f5c8c10dbb0 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8c1a5f70 con 0x7f5c8c10dbb0 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3653624371 0 0) 0x7f5c74003520 con 0x7f5c8c10a040 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c8c1a7190 con 0x7f5c8c10a040 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f5c80003f40 con 0x7f5c8c10a040 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3449391233 0 0) 0x7f5c8c1a83d0 con 0x7f5c8c0ffbd0 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c74003520 con 0x7f5c8c0ffbd0 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f5c7c004530 con 0x7f5c8c0ffbd0 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1730252927 0 0) 0x7f5c8c1a7190 con 0x7f5c8c10a040 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 >> v1:172.21.15.151:6789/0 conn(0x7f5c8c10dbb0 legacy=0x7f5c8c1a4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 >> v1:172.21.15.94:6789/0 conn(0x7f5c8c0ffbd0 legacy=0x7f5c8c106510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c1a9610 con 0x7f5c8c10a040 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c928ea640 1 -- 172.21.15.94:0/1446764348 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c1a61a0 con 0x7f5c8c10a040 2024-03-12T03:47:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c928ea640 1 -- 172.21.15.94:0/1446764348 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c1a66f0 con 0x7f5c8c10a040 2024-03-12T03:47:06.147 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f5c80001360 con 0x7f5c8c10a040 2024-03-12T03:47:06.147 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.141+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f5c80005b50 con 0x7f5c8c10a040 2024-03-12T03:47:06.147 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.142+0000 7f5c928ea640 1 -- 172.21.15.94:0/1446764348 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c78005080 con 0x7f5c8c10a040 2024-03-12T03:47:06.147 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.142+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f5c80000ed0 con 0x7f5c8c10a040 2024-03-12T03:47:06.147 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.143+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(797..797 src has 254..797) v4 ==== 7817+0+0 (unknown 487389545 0 0) 0x7f5c80089580 con 0x7f5c8c10a040 2024-03-12T03:47:06.150 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.148+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f5c80005b30 con 0x7f5c8c10a040 2024-03-12T03:47:06.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:05 smithi094 ceph-mon[40484]: osdmap e796: 8 total, 8 up, 8 in 2024-03-12T03:47:06.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:05 smithi094 ceph-mon[40484]: pgmap v1711: 164 pgs: 164 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:06.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:05 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3299069758' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pg_num": 12}]: dispatch 2024-03-12T03:47:06.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:06.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:05 smithi094 ceph-mon[29291]: osdmap e796: 8 total, 8 up, 8 in 2024-03-12T03:47:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:05 smithi094 ceph-mon[29291]: pgmap v1711: 164 pgs: 164 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:05 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3299069758' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pg_num": 12}]: dispatch 2024-03-12T03:47:06.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:06.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:05 smithi151 ceph-mon[32923]: osdmap e796: 8 total, 8 up, 8 in 2024-03-12T03:47:06.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:05 smithi151 ceph-mon[32923]: pgmap v1711: 164 pgs: 164 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:06.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:05 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3299069758' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pg_num": 12}]: dispatch 2024-03-12T03:47:06.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:06.327 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.325+0000 7f5c928ea640 1 -- 172.21.15.94:0/1446764348 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f5c78005370 con 0x7f5c8c10a040 2024-03-12T03:47:06.950 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:06.947+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v798) v1 ==== 223+0+0 (unknown 1423444025 0 0) 0x7f5c80056810 con 0x7f5c8c10a040 2024-03-12T03:47:07.063 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:07.060+0000 7f5c928ea640 1 -- 172.21.15.94:0/1446764348 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f5c78001090 con 0x7f5c8c10a040 2024-03-12T03:47:07.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:06 smithi094 ceph-mon[29291]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:47:07.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3299069758' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pg_num": 12}]': finished 2024-03-12T03:47:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:06 smithi094 ceph-mon[29291]: osdmap e797: 8 total, 8 up, 8 in 2024-03-12T03:47:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3299069758' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pg_num": 12}]: dispatch 2024-03-12T03:47:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:06 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1446764348' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:47:07.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:06 smithi094 ceph-mon[29291]: from='client.57173 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:47:07.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:06 smithi094 ceph-mon[40484]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:47:07.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3299069758' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pg_num": 12}]': finished 2024-03-12T03:47:07.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:06 smithi094 ceph-mon[40484]: osdmap e797: 8 total, 8 up, 8 in 2024-03-12T03:47:07.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3299069758' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pg_num": 12}]: dispatch 2024-03-12T03:47:07.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:06 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1446764348' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:47:07.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:06 smithi094 ceph-mon[40484]: from='client.57173 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:47:07.217 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:06 smithi151 ceph-mon[32923]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:47:07.217 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3299069758' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pg_num": 12}]': finished 2024-03-12T03:47:07.217 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:06 smithi151 ceph-mon[32923]: osdmap e797: 8 total, 8 up, 8 in 2024-03-12T03:47:07.218 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3299069758' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pg_num": 12}]: dispatch 2024-03-12T03:47:07.218 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:06 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1446764348' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:47:07.218 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:06 smithi151 ceph-mon[32923]: from='client.57173 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:47:07.957 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:07.955+0000 7f5c897fa640 1 -- 172.21.15.94:0/1446764348 <== mon.2 v1:172.21.15.94:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v799) v1 ==== 223+0+0 (unknown 496872777 0 0) 0x7f5c8005a260 con 0x7f5c8c10a040 2024-03-12T03:47:07.957 INFO:tasks.workunit.client.0.smithi094.stderr:set-quota max_objects = 10 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 2024-03-12T03:47:07.960 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:07.957+0000 7f5c928ea640 1 -- 172.21.15.94:0/1446764348 >> v1:172.21.15.94:6800/2099808905 conn(0x7f5c7406e9a0 legacy=0x7f5c74070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:07.960 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:07.957+0000 7f5c928ea640 1 -- 172.21.15.94:0/1446764348 >> v1:172.21.15.94:6790/0 conn(0x7f5c8c10a040 legacy=0x7f5c8c1a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:07.960 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:07.958+0000 7f5c928ea640 1 -- 172.21.15.94:0/1446764348 shutdown_connections 2024-03-12T03:47:07.960 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:07.958+0000 7f5c928ea640 1 -- 172.21.15.94:0/1446764348 >> 172.21.15.94:0/1446764348 conn(0x7f5c8c075350 msgr2=0x7f5c8c1110b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:47:07.960 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:07.958+0000 7f5c928ea640 1 -- 172.21.15.94:0/1446764348 shutdown_connections 2024-03-12T03:47:07.960 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:07.958+0000 7f5c928ea640 1 -- 172.21.15.94:0/1446764348 wait complete. 2024-03-12T03:47:07.971 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool application enable f98a828b-0804-4d73-9e78-2d8570d1ebe3 rados 2024-03-12T03:47:08.061 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.054+0000 7f2ba3892640 1 -- 172.21.15.94:0/1327811731 >> v1:172.21.15.94:6789/0 conn(0x7f2b9c1057c0 legacy=0x7f2b9c105ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:08.061 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.054+0000 7f2ba3892640 1 -- 172.21.15.94:0/1327811731 shutdown_connections 2024-03-12T03:47:08.061 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.054+0000 7f2ba3892640 1 -- 172.21.15.94:0/1327811731 >> 172.21.15.94:0/1327811731 conn(0x7f2b9c075320 msgr2=0x7f2b9c075730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:47:08.061 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.055+0000 7f2ba3892640 1 -- 172.21.15.94:0/1327811731 shutdown_connections 2024-03-12T03:47:08.061 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.055+0000 7f2ba3892640 1 -- 172.21.15.94:0/1327811731 wait complete. 2024-03-12T03:47:08.061 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.055+0000 7f2ba3892640 1 Processor -- start 2024-03-12T03:47:08.061 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.055+0000 7f2ba3892640 1 -- start start 2024-03-12T03:47:08.061 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.055+0000 7f2ba3892640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2b9c1a4e70 con 0x7f2b9c10c230 2024-03-12T03:47:08.062 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.055+0000 7f2ba3892640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2b9c1a60b0 con 0x7f2b9c1057c0 2024-03-12T03:47:08.062 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.055+0000 7f2ba3892640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2b9c1a72f0 con 0x7f2b9c108660 2024-03-12T03:47:08.062 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.055+0000 7f2ba0e2d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f2b9c108660 0x7f2b9c0fdf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:60914/0 (socket says 172.21.15.94:60914) 2024-03-12T03:47:08.062 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.055+0000 7f2ba1e2f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2b9c10c230 0x7f2b9c1a3690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43714/0 (socket says 172.21.15.94:43714) 2024-03-12T03:47:08.062 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.055+0000 7f2ba0e2d640 1 -- 172.21.15.94:0/3007911633 learned_addr learned my addr 172.21.15.94:0/3007911633 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:47:08.062 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810008387 0 0) 0x7f2b9c1a72f0 con 0x7f2b9c108660 2024-03-12T03:47:08.062 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b88003160 con 0x7f2b9c108660 2024-03-12T03:47:08.062 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168183327 0 0) 0x7f2b9c1a4e70 con 0x7f2b9c10c230 2024-03-12T03:47:08.062 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b9c1a72f0 con 0x7f2b9c10c230 2024-03-12T03:47:08.062 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329714412 0 0) 0x7f2b9c1a60b0 con 0x7f2b9c1057c0 2024-03-12T03:47:08.062 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b9c1a4e70 con 0x7f2b9c1057c0 2024-03-12T03:47:08.063 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2466899044 0 0) 0x7f2b88003160 con 0x7f2b9c108660 2024-03-12T03:47:08.063 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b9c1a60b0 con 0x7f2b9c108660 2024-03-12T03:47:08.063 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f2b8c0018d0 con 0x7f2b9c108660 2024-03-12T03:47:08.063 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3324373075 0 0) 0x7f2b9c1a72f0 con 0x7f2b9c10c230 2024-03-12T03:47:08.063 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b88003160 con 0x7f2b9c10c230 2024-03-12T03:47:08.063 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f2b94001c10 con 0x7f2b9c10c230 2024-03-12T03:47:08.063 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2601563428 0 0) 0x7f2b9c1a4e70 con 0x7f2b9c1057c0 2024-03-12T03:47:08.063 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b9c1a72f0 con 0x7f2b9c1057c0 2024-03-12T03:47:08.063 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f2b98004430 con 0x7f2b9c1057c0 2024-03-12T03:47:08.063 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2343645470 0 0) 0x7f2b9c1a60b0 con 0x7f2b9c108660 2024-03-12T03:47:08.063 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 >> v1:172.21.15.151:6789/0 conn(0x7f2b9c1057c0 legacy=0x7f2b9c0fd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:08.064 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 >> v1:172.21.15.94:6789/0 conn(0x7f2b9c10c230 legacy=0x7f2b9c1a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:08.064 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c1a8530 con 0x7f2b9c108660 2024-03-12T03:47:08.064 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2ba3892640 1 -- 172.21.15.94:0/3007911633 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9c1a7520 con 0x7f2b9c108660 2024-03-12T03:47:08.064 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2ba3892640 1 -- 172.21.15.94:0/3007911633 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9c1a79f0 con 0x7f2b9c108660 2024-03-12T03:47:08.064 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f2b8c003aa0 con 0x7f2b9c108660 2024-03-12T03:47:08.064 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.056+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f2b8c005b70 con 0x7f2b9c108660 2024-03-12T03:47:08.064 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.057+0000 7f2ba3892640 1 -- 172.21.15.94:0/3007911633 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b80005080 con 0x7f2b9c108660 2024-03-12T03:47:08.064 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.058+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f2b8c003610 con 0x7f2b9c108660 2024-03-12T03:47:08.064 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.058+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(799..799 src has 254..799) v4 ==== 7817+0+0 (unknown 439928799 0 0) 0x7f2b8c0896a0 con 0x7f2b9c108660 2024-03-12T03:47:08.065 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.062+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f2b8c056930 con 0x7f2b9c108660 2024-03-12T03:47:08.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:07 smithi094 ceph-mon[29291]: from='client.57173 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]': finished 2024-03-12T03:47:08.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:07 smithi094 ceph-mon[29291]: osdmap e798: 8 total, 8 up, 8 in 2024-03-12T03:47:08.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:07 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1446764348' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:47:08.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:07 smithi094 ceph-mon[29291]: from='client.57173 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:47:08.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:07 smithi094 ceph-mon[29291]: pgmap v1714: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:47:08.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:07 smithi094 ceph-mon[40484]: from='client.57173 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]': finished 2024-03-12T03:47:08.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:07 smithi094 ceph-mon[40484]: osdmap e798: 8 total, 8 up, 8 in 2024-03-12T03:47:08.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:07 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1446764348' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:47:08.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:07 smithi094 ceph-mon[40484]: from='client.57173 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:47:08.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:07 smithi094 ceph-mon[40484]: pgmap v1714: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:47:08.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:07 smithi151 ceph-mon[32923]: from='client.57173 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]': finished 2024-03-12T03:47:08.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:07 smithi151 ceph-mon[32923]: osdmap e798: 8 total, 8 up, 8 in 2024-03-12T03:47:08.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:07 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1446764348' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:47:08.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:07 smithi151 ceph-mon[32923]: from='client.57173 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:47:08.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:07 smithi151 ceph-mon[32923]: pgmap v1714: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 834 MiB used, 714 GiB / 715 GiB avail 2024-03-12T03:47:08.245 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.242+0000 7f2ba3892640 1 -- 172.21.15.94:0/3007911633 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"} v 0) v1 -- 0x7f2b80005370 con 0x7f2b9c108660 2024-03-12T03:47:08.958 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:08.955+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]=0 enabled application 'rados' on pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' v800) v1 ==== 213+0+0 (unknown 2271804872 0 0) 0x7f2b8c05a380 con 0x7f2b9c108660 2024-03-12T03:47:09.069 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:09.066+0000 7f2ba3892640 1 -- 172.21.15.94:0/3007911633 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"} v 0) v1 -- 0x7f2b80005b80 con 0x7f2b9c108660 2024-03-12T03:47:09.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:08 smithi094 ceph-mon[29291]: from='client.57173 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]': finished 2024-03-12T03:47:09.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:08 smithi094 ceph-mon[29291]: osdmap e799: 8 total, 8 up, 8 in 2024-03-12T03:47:09.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:08 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3007911633' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]: dispatch 2024-03-12T03:47:09.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:08 smithi094 ceph-mon[29291]: from='client.57179 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]: dispatch 2024-03-12T03:47:09.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:08 smithi094 ceph-mon[40484]: from='client.57173 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]': finished 2024-03-12T03:47:09.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:08 smithi094 ceph-mon[40484]: osdmap e799: 8 total, 8 up, 8 in 2024-03-12T03:47:09.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:08 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3007911633' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]: dispatch 2024-03-12T03:47:09.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:08 smithi094 ceph-mon[40484]: from='client.57179 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]: dispatch 2024-03-12T03:47:09.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:08 smithi151 ceph-mon[32923]: from='client.57173 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "10"}]': finished 2024-03-12T03:47:09.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:08 smithi151 ceph-mon[32923]: osdmap e799: 8 total, 8 up, 8 in 2024-03-12T03:47:09.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:08 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3007911633' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]: dispatch 2024-03-12T03:47:09.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:08 smithi151 ceph-mon[32923]: from='client.57179 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]: dispatch 2024-03-12T03:47:09.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:47:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:47:09.968 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:09.966+0000 7f2b927fc640 1 -- 172.21.15.94:0/3007911633 <== mon.2 v1:172.21.15.94:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]=0 enabled application 'rados' on pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' v801) v1 ==== 213+0+0 (unknown 3175680814 0 0) 0x7f2b8c05e9d0 con 0x7f2b9c108660 2024-03-12T03:47:09.969 INFO:tasks.workunit.client.0.smithi094.stderr:enabled application 'rados' on pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' 2024-03-12T03:47:09.971 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:09.968+0000 7f2ba3892640 1 -- 172.21.15.94:0/3007911633 >> v1:172.21.15.94:6800/2099808905 conn(0x7f2b8806ed30 legacy=0x7f2b880711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:09.971 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:09.968+0000 7f2ba3892640 1 -- 172.21.15.94:0/3007911633 >> v1:172.21.15.94:6790/0 conn(0x7f2b9c108660 legacy=0x7f2b9c0fdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:09.971 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:09.968+0000 7f2ba3892640 1 -- 172.21.15.94:0/3007911633 shutdown_connections 2024-03-12T03:47:09.971 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:09.968+0000 7f2ba3892640 1 -- 172.21.15.94:0/3007911633 >> 172.21.15.94:0/3007911633 conn(0x7f2b9c075320 msgr2=0x7f2b9c10e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:47:09.971 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:09.968+0000 7f2ba3892640 1 -- 172.21.15.94:0/3007911633 shutdown_connections 2024-03-12T03:47:09.971 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:09.968+0000 7f2ba3892640 1 -- 172.21.15.94:0/3007911633 wait complete. 2024-03-12T03:47:09.982 INFO:tasks.workunit.client.0.smithi094.stderr:++ seq 1 10 2024-03-12T03:47:09.993 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:47:09.994 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put obj1 /etc/passwd 2024-03-12T03:47:10.029 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:47:10.029 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put obj2 /etc/passwd 2024-03-12T03:47:10.064 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:47:10.064 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put obj3 /etc/passwd 2024-03-12T03:47:10.097 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:47:10.097 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put obj4 /etc/passwd 2024-03-12T03:47:10.129 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:47:10.129 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put obj5 /etc/passwd 2024-03-12T03:47:10.162 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:47:10.162 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put obj6 /etc/passwd 2024-03-12T03:47:10.195 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:47:10.195 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put obj7 /etc/passwd 2024-03-12T03:47:10.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:09 smithi151 ceph-mon[32923]: from='client.57179 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]': finished 2024-03-12T03:47:10.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:09 smithi151 ceph-mon[32923]: osdmap e800: 8 total, 8 up, 8 in 2024-03-12T03:47:10.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:09 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3007911633' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]: dispatch 2024-03-12T03:47:10.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:09 smithi151 ceph-mon[32923]: from='client.57179 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]: dispatch 2024-03-12T03:47:10.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:09 smithi151 ceph-mon[32923]: pgmap v1717: 176 pgs: 176 active+clean; 582 KiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:10.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:09 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:10.227 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:47:10.227 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put obj8 /etc/passwd 2024-03-12T03:47:10.259 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:47:10.260 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put obj9 /etc/passwd 2024-03-12T03:47:10.291 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:47:10.292 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put obj10 /etc/passwd 2024-03-12T03:47:10.323 INFO:tasks.workunit.client.0.smithi094.stderr:+ sleep 30 2024-03-12T03:47:10.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:09 smithi094 ceph-mon[29291]: from='client.57179 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]': finished 2024-03-12T03:47:10.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:09 smithi094 ceph-mon[29291]: osdmap e800: 8 total, 8 up, 8 in 2024-03-12T03:47:10.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:09 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3007911633' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]: dispatch 2024-03-12T03:47:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:09 smithi094 ceph-mon[29291]: from='client.57179 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]: dispatch 2024-03-12T03:47:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:09 smithi094 ceph-mon[29291]: pgmap v1717: 176 pgs: 176 active+clean; 582 KiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:10.460 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:09 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:10.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:09 smithi094 ceph-mon[40484]: from='client.57179 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]': finished 2024-03-12T03:47:10.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:09 smithi094 ceph-mon[40484]: osdmap e800: 8 total, 8 up, 8 in 2024-03-12T03:47:10.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:09 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3007911633' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]: dispatch 2024-03-12T03:47:10.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:09 smithi094 ceph-mon[40484]: from='client.57179 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]: dispatch 2024-03-12T03:47:10.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:09 smithi094 ceph-mon[40484]: pgmap v1717: 176 pgs: 176 active+clean; 582 KiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:10.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:09 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:10.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:47:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:47:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:47:11.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:10 smithi094 ceph-mon[29291]: from='client.57179 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]': finished 2024-03-12T03:47:11.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:10 smithi094 ceph-mon[29291]: osdmap e801: 8 total, 8 up, 8 in 2024-03-12T03:47:11.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:10 smithi094 ceph-mon[40484]: from='client.57179 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]': finished 2024-03-12T03:47:11.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:10 smithi094 ceph-mon[40484]: osdmap e801: 8 total, 8 up, 8 in 2024-03-12T03:47:11.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:10 smithi151 ceph-mon[32923]: from='client.57179 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "app": "rados"}]': finished 2024-03-12T03:47:11.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:10 smithi151 ceph-mon[32923]: osdmap e801: 8 total, 8 up, 8 in 2024-03-12T03:47:12.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:11 smithi094 ceph-mon[29291]: pgmap v1719: 176 pgs: 176 active+clean; 582 KiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:12.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:11 smithi094 ceph-mon[40484]: pgmap v1719: 176 pgs: 176 active+clean; 582 KiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:12.464 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:11 smithi151 ceph-mon[32923]: pgmap v1719: 176 pgs: 176 active+clean; 582 KiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:14.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:14 smithi094 ceph-mon[29291]: pgmap v1720: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 20 KiB/s wr, 3 op/s 2024-03-12T03:47:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:14 smithi094 ceph-mon[29291]: pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' is full (reached quota's max_objects: 10) 2024-03-12T03:47:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:14 smithi094 ceph-mon[29291]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-12T03:47:14.710 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:14 smithi094 ceph-mon[29291]: osdmap e802: 8 total, 8 up, 8 in 2024-03-12T03:47:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:14 smithi094 ceph-mon[40484]: pgmap v1720: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 20 KiB/s wr, 3 op/s 2024-03-12T03:47:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:14 smithi094 ceph-mon[40484]: pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' is full (reached quota's max_objects: 10) 2024-03-12T03:47:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:14 smithi094 ceph-mon[40484]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-12T03:47:14.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:14 smithi094 ceph-mon[40484]: osdmap e802: 8 total, 8 up, 8 in 2024-03-12T03:47:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:14 smithi151 ceph-mon[32923]: pgmap v1720: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 20 KiB/s wr, 3 op/s 2024-03-12T03:47:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:14 smithi151 ceph-mon[32923]: pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' is full (reached quota's max_objects: 10) 2024-03-12T03:47:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:14 smithi151 ceph-mon[32923]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-12T03:47:14.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:14 smithi151 ceph-mon[32923]: osdmap e802: 8 total, 8 up, 8 in 2024-03-12T03:47:16.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:16 smithi094 ceph-mon[29291]: pgmap v1722: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 820 B/s rd, 19 KiB/s wr, 2 op/s 2024-03-12T03:47:16.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:16 smithi094 ceph-mon[40484]: pgmap v1722: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 820 B/s rd, 19 KiB/s wr, 2 op/s 2024-03-12T03:47:16.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:16 smithi151 ceph-mon[32923]: pgmap v1722: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 820 B/s rd, 19 KiB/s wr, 2 op/s 2024-03-12T03:47:18.517 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:18 smithi151 ceph-mon[32923]: pgmap v1723: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-12T03:47:18.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:18 smithi094 ceph-mon[29291]: pgmap v1723: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-12T03:47:18.710 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:18 smithi094 ceph-mon[40484]: pgmap v1723: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-12T03:47:19.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:47:19 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:47:20.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:20 smithi094 ceph-mon[29291]: pgmap v1724: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-03-12T03:47:20.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:20.512 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:20 smithi094 ceph-mon[40484]: pgmap v1724: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-03-12T03:47:20.512 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:20.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:20 smithi151 ceph-mon[32923]: pgmap v1724: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-03-12T03:47:20.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:20.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:47:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:47:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:47:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:21 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:21 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:21 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:21.961 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:21 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:21 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:21 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:22.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:22 smithi094 ceph-mon[40484]: pgmap v1725: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-12T03:47:22.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:22 smithi094 ceph-mon[29291]: pgmap v1725: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-12T03:47:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:22 smithi151 ceph-mon[32923]: pgmap v1725: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-12T03:47:24.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:24 smithi094 ceph-mon[29291]: pgmap v1726: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:47:24.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:24 smithi094 ceph-mon[40484]: pgmap v1726: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:47:24.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:24 smithi151 ceph-mon[32923]: pgmap v1726: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:47:26.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:26 smithi094 ceph-mon[29291]: pgmap v1727: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 874 B/s rd, 0 op/s 2024-03-12T03:47:26.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:26 smithi094 ceph-mon[40484]: pgmap v1727: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 874 B/s rd, 0 op/s 2024-03-12T03:47:26.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:26 smithi151 ceph-mon[32923]: pgmap v1727: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 874 B/s rd, 0 op/s 2024-03-12T03:47:28.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:28 smithi094 ceph-mon[29291]: pgmap v1728: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:28.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:28 smithi094 ceph-mon[40484]: pgmap v1728: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:28.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:28 smithi151 ceph-mon[32923]: pgmap v1728: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:29.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:47:29 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:47:30.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:30 smithi094 ceph-mon[29291]: pgmap v1729: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:30.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:47:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:47:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:47:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:30 smithi094 ceph-mon[40484]: pgmap v1729: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:30 smithi151 ceph-mon[32923]: pgmap v1729: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:30.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:32.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:32 smithi094 ceph-mon[29291]: pgmap v1730: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:32.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:32 smithi094 ceph-mon[40484]: pgmap v1730: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:32.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:32 smithi151 ceph-mon[32923]: pgmap v1730: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:34.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:34 smithi094 ceph-mon[29291]: pgmap v1731: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:34.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:34 smithi094 ceph-mon[40484]: pgmap v1731: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:34.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:34 smithi151 ceph-mon[32923]: pgmap v1731: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:35.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:35.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:35.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:36.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:36 smithi094 ceph-mon[29291]: pgmap v1732: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:36.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:36 smithi094 ceph-mon[40484]: pgmap v1732: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:36.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:36 smithi151 ceph-mon[32923]: pgmap v1732: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:38.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:38 smithi094 ceph-mon[29291]: pgmap v1733: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:38.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:38 smithi094 ceph-mon[40484]: pgmap v1733: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:38.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:38 smithi151 ceph-mon[32923]: pgmap v1733: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:39.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:47:39 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:47:40.325 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=154243 2024-03-12T03:47:40.325 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool set-quota f98a828b-0804-4d73-9e78-2d8570d1ebe3 max_objects 100 2024-03-12T03:47:40.325 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put onemore /etc/passwd 2024-03-12T03:47:40.418 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.415+0000 7fcaa4959640 1 -- 172.21.15.94:0/2671297147 >> v1:172.21.15.94:6789/0 conn(0x7fcaa0102390 legacy=0x7fcaa0102770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:40.418 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.415+0000 7fcaa4959640 1 -- 172.21.15.94:0/2671297147 shutdown_connections 2024-03-12T03:47:40.418 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.415+0000 7fcaa4959640 1 -- 172.21.15.94:0/2671297147 >> 172.21.15.94:0/2671297147 conn(0x7fcaa00fbaf0 msgr2=0x7fcaa00fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:47:40.418 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.415+0000 7fcaa4959640 1 -- 172.21.15.94:0/2671297147 shutdown_connections 2024-03-12T03:47:40.418 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.415+0000 7fcaa4959640 1 -- 172.21.15.94:0/2671297147 wait complete. 2024-03-12T03:47:40.422 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.416+0000 7fcaa4959640 1 Processor -- start 2024-03-12T03:47:40.422 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.416+0000 7fcaa4959640 1 -- start start 2024-03-12T03:47:40.422 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fcaa4959640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcaa01a6130 con 0x7fcaa010a190 2024-03-12T03:47:40.422 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fcaa4959640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcaa01a7370 con 0x7fcaa0102390 2024-03-12T03:47:40.422 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fcaa4959640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcaa01a85b0 con 0x7fcaa010dca0 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca95d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcaa010a190 0x7fcaa01a0ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57558/0 (socket says 172.21.15.94:57558) 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca95d9b640 1 -- 172.21.15.94:0/2417958138 learned_addr learned my addr 172.21.15.94:0/2417958138 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca9ed9d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fcaa010dca0 0x7fcaa01a4830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:49834/0 (socket says 172.21.15.94:49834) 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96890305 0 0) 0x7fcaa01a85b0 con 0x7fcaa010dca0 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca8c003520 con 0x7fcaa010dca0 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821678339 0 0) 0x7fcaa01a6130 con 0x7fcaa010a190 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa01a85b0 con 0x7fcaa010a190 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676735858 0 0) 0x7fcaa01a7370 con 0x7fcaa0102390 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa01a6130 con 0x7fcaa0102390 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3957248456 0 0) 0x7fca8c003520 con 0x7fcaa010dca0 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcaa01a7370 con 0x7fcaa010dca0 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fca98001c10 con 0x7fcaa010dca0 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 559968488 0 0) 0x7fcaa01a85b0 con 0x7fcaa010a190 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca8c003520 con 0x7fcaa010a190 2024-03-12T03:47:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fca88001850 con 0x7fcaa010a190 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2038994743 0 0) 0x7fcaa01a6130 con 0x7fcaa0102390 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcaa01a85b0 con 0x7fcaa0102390 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fca90001880 con 0x7fcaa0102390 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2273295782 0 0) 0x7fcaa01a7370 con 0x7fcaa010dca0 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 >> v1:172.21.15.151:6789/0 conn(0x7fcaa0102390 legacy=0x7fcaa0109970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 >> v1:172.21.15.94:6789/0 conn(0x7fcaa010a190 legacy=0x7fcaa01a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.417+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa01a97f0 con 0x7fcaa010dca0 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.418+0000 7fcaa4959640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa01a6360 con 0x7fcaa010dca0 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.418+0000 7fcaa4959640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa01a6830 con 0x7fcaa010dca0 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.418+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fca98002000 con 0x7fcaa010dca0 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.418+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fca98005fe0 con 0x7fcaa010dca0 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.419+0000 7fcaa4959640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca80005080 con 0x7fcaa010dca0 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.419+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7fca98003c40 con 0x7fcaa010dca0 2024-03-12T03:47:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.420+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(802..802 src has 254..802) v4 ==== 7830+0+0 (unknown 3099464445 0 0) 0x7fca98089b10 con 0x7fcaa010dca0 2024-03-12T03:47:40.425 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.420+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=803}) v3 -- 0x7fcaa01a7370 con 0x7fcaa010dca0 2024-03-12T03:47:40.426 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.424+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fca98056d90 con 0x7fcaa010dca0 2024-03-12T03:47:40.602 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.599+0000 7fcaa4959640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7fca80005370 con 0x7fcaa010dca0 2024-03-12T03:47:40.625 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.623+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.2 v1:172.21.15.94:6790/0 10 ==== osd_map(803..803 src has 254..803) v4 ==== 628+0+0 (unknown 2613559655 0 0) 0x7fca98087900 con 0x7fcaa010dca0 2024-03-12T03:47:40.625 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.623+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=804}) v3 -- 0x7fcaa01a6130 con 0x7fcaa010dca0 2024-03-12T03:47:40.628 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.626+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.2 v1:172.21.15.94:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v803) v1 ==== 225+0+0 (unknown 2915176057 0 0) 0x7fca9805a7e0 con 0x7fcaa010dca0 2024-03-12T03:47:40.735 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:40.733+0000 7fcaa4959640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7fca80005b80 con 0x7fcaa010dca0 2024-03-12T03:47:40.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:47:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:47:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:47:40.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:40 smithi094 ceph-mon[40484]: pgmap v1734: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:40 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2417958138' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]: dispatch 2024-03-12T03:47:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:40 smithi094 ceph-mon[40484]: from='client.57284 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]: dispatch 2024-03-12T03:47:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:40 smithi094 ceph-mon[29291]: pgmap v1734: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:40 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2417958138' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]: dispatch 2024-03-12T03:47:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:40 smithi094 ceph-mon[29291]: from='client.57284 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]: dispatch 2024-03-12T03:47:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:40 smithi151 ceph-mon[32923]: pgmap v1734: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:40 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2417958138' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]: dispatch 2024-03-12T03:47:40.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:40 smithi151 ceph-mon[32923]: from='client.57284 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]: dispatch 2024-03-12T03:47:41.637 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:41.634+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.2 v1:172.21.15.94:6790/0 12 ==== osd_map(804..804 src has 254..804) v4 ==== 628+0+0 (unknown 1607388942 0 0) 0x7fca980526d0 con 0x7fcaa010dca0 2024-03-12T03:47:41.637 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:41.634+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=805}) v3 -- 0x7fca8c07bd20 con 0x7fcaa010dca0 2024-03-12T03:47:41.640 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:41.638+0000 7fca97fff640 1 -- 172.21.15.94:0/2417958138 <== mon.2 v1:172.21.15.94:6790/0 13 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v804) v1 ==== 225+0+0 (unknown 2003548945 0 0) 0x7fca9805ee30 con 0x7fcaa010dca0 2024-03-12T03:47:41.641 INFO:tasks.workunit.client.0.smithi094.stderr:set-quota max_objects = 100 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 2024-03-12T03:47:41.644 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:41.642+0000 7fcaa4959640 1 -- 172.21.15.94:0/2417958138 >> v1:172.21.15.94:6800/2099808905 conn(0x7fca8c06ed30 legacy=0x7fca8c0711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:41.644 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:41.642+0000 7fcaa4959640 1 -- 172.21.15.94:0/2417958138 >> v1:172.21.15.94:6790/0 conn(0x7fcaa010dca0 legacy=0x7fcaa01a4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:41.645 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:41.642+0000 7fcaa4959640 1 -- 172.21.15.94:0/2417958138 shutdown_connections 2024-03-12T03:47:41.645 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:41.642+0000 7fcaa4959640 1 -- 172.21.15.94:0/2417958138 >> 172.21.15.94:0/2417958138 conn(0x7fcaa00fbaf0 msgr2=0x7fcaa00fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:47:41.645 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:41.643+0000 7fcaa4959640 1 -- 172.21.15.94:0/2417958138 shutdown_connections 2024-03-12T03:47:41.645 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:41.643+0000 7fcaa4959640 1 -- 172.21.15.94:0/2417958138 wait complete. 2024-03-12T03:47:41.654 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 154243 2024-03-12T03:47:41.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:41 smithi094 ceph-mon[40484]: from='client.57284 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]': finished 2024-03-12T03:47:41.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:41 smithi094 ceph-mon[40484]: osdmap e803: 8 total, 8 up, 8 in 2024-03-12T03:47:41.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:41 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/2417958138' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]: dispatch 2024-03-12T03:47:41.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:41 smithi094 ceph-mon[40484]: from='client.57284 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]: dispatch 2024-03-12T03:47:41.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:41 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:47:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:41 smithi094 ceph-mon[29291]: from='client.57284 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]': finished 2024-03-12T03:47:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:41 smithi094 ceph-mon[29291]: osdmap e803: 8 total, 8 up, 8 in 2024-03-12T03:47:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:41 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/2417958138' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]: dispatch 2024-03-12T03:47:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:41 smithi094 ceph-mon[29291]: from='client.57284 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]: dispatch 2024-03-12T03:47:41.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:41 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:47:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:41 smithi151 ceph-mon[32923]: from='client.57284 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]': finished 2024-03-12T03:47:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:41 smithi151 ceph-mon[32923]: osdmap e803: 8 total, 8 up, 8 in 2024-03-12T03:47:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:41 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/2417958138' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]: dispatch 2024-03-12T03:47:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:41 smithi151 ceph-mon[32923]: from='client.57284 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]: dispatch 2024-03-12T03:47:41.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:41 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:47:42.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:42 smithi094 ceph-mon[29291]: pgmap v1736: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:47:42.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:42 smithi094 ceph-mon[29291]: from='client.57284 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]': finished 2024-03-12T03:47:42.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:42 smithi094 ceph-mon[29291]: osdmap e804: 8 total, 8 up, 8 in 2024-03-12T03:47:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:42 smithi094 ceph-mon[40484]: pgmap v1736: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:47:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:42 smithi094 ceph-mon[40484]: from='client.57284 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]': finished 2024-03-12T03:47:42.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:42 smithi094 ceph-mon[40484]: osdmap e804: 8 total, 8 up, 8 in 2024-03-12T03:47:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:42 smithi151 ceph-mon[32923]: pgmap v1736: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:47:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:42 smithi151 ceph-mon[32923]: from='client.57284 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "100"}]': finished 2024-03-12T03:47:42.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:42 smithi151 ceph-mon[32923]: osdmap e804: 8 total, 8 up, 8 in 2024-03-12T03:47:43.497 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 0 -ne 0 ']' 2024-03-12T03:47:43.497 INFO:tasks.workunit.client.0.smithi094.stderr:+ true 2024-03-12T03:47:43.497 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put twomore /etc/passwd 2024-03-12T03:47:43.530 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool set-quota f98a828b-0804-4d73-9e78-2d8570d1ebe3 max_bytes 100 2024-03-12T03:47:43.624 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.621+0000 7fbef0384640 1 -- 172.21.15.94:0/1313097464 >> v1:172.21.15.94:6790/0 conn(0x7fbee810dc30 legacy=0x7fbee81100c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:43.624 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.621+0000 7fbef0384640 1 -- 172.21.15.94:0/1313097464 shutdown_connections 2024-03-12T03:47:43.624 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.621+0000 7fbef0384640 1 -- 172.21.15.94:0/1313097464 >> 172.21.15.94:0/1313097464 conn(0x7fbee80fbb40 msgr2=0x7fbee80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:47:43.624 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.621+0000 7fbef0384640 1 -- 172.21.15.94:0/1313097464 shutdown_connections 2024-03-12T03:47:43.624 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.621+0000 7fbef0384640 1 -- 172.21.15.94:0/1313097464 wait complete. 2024-03-12T03:47:43.625 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.622+0000 7fbef0384640 1 Processor -- start 2024-03-12T03:47:43.625 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.622+0000 7fbef0384640 1 -- start start 2024-03-12T03:47:43.626 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.622+0000 7fbef0384640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbee810d580 con 0x7fbee810a120 2024-03-12T03:47:43.627 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.622+0000 7fbef0384640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbee81a9710 con 0x7fbee8102320 2024-03-12T03:47:43.627 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.622+0000 7fbef0384640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbee81aa8f0 con 0x7fbee810dc30 2024-03-12T03:47:43.627 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.622+0000 7fbeed91f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbee810a120 0x7fbee81a38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57592/0 (socket says 172.21.15.94:57592) 2024-03-12T03:47:43.627 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.622+0000 7fbeee921640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fbee810dc30 0x7fbee81a6fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:49896/0 (socket says 172.21.15.94:49896) 2024-03-12T03:47:43.627 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.622+0000 7fbeed91f640 1 -- 172.21.15.94:0/3347396902 learned_addr learned my addr 172.21.15.94:0/3347396902 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:47:43.627 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.622+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005551056 0 0) 0x7fbee810d580 con 0x7fbee810a120 2024-03-12T03:47:43.627 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed4003520 con 0x7fbee810a120 2024-03-12T03:47:43.627 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712986180 0 0) 0x7fbee81aa8f0 con 0x7fbee810dc30 2024-03-12T03:47:43.627 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee810d580 con 0x7fbee810dc30 2024-03-12T03:47:43.627 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353866842 0 0) 0x7fbee81a9710 con 0x7fbee8102320 2024-03-12T03:47:43.627 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee81aa8f0 con 0x7fbee8102320 2024-03-12T03:47:43.627 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1340364113 0 0) 0x7fbed4003520 con 0x7fbee810a120 2024-03-12T03:47:43.627 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbee81a9710 con 0x7fbee810a120 2024-03-12T03:47:43.628 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fbed8001a30 con 0x7fbee810a120 2024-03-12T03:47:43.628 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 530668436 0 0) 0x7fbee810d580 con 0x7fbee810dc30 2024-03-12T03:47:43.628 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbed4003520 con 0x7fbee810dc30 2024-03-12T03:47:43.628 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fbee0004720 con 0x7fbee810dc30 2024-03-12T03:47:43.628 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2248042242 0 0) 0x7fbee81a9710 con 0x7fbee810a120 2024-03-12T03:47:43.628 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 >> v1:172.21.15.94:6790/0 conn(0x7fbee810dc30 legacy=0x7fbee81a6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:43.628 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 >> v1:172.21.15.151:6789/0 conn(0x7fbee8102320 legacy=0x7fbee810c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:43.628 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee81abb30 con 0x7fbee810a120 2024-03-12T03:47:43.628 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbef0384640 1 -- 172.21.15.94:0/3347396902 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee81aab20 con 0x7fbee810a120 2024-03-12T03:47:43.628 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbef0384640 1 -- 172.21.15.94:0/3347396902 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee81aaff0 con 0x7fbee810a120 2024-03-12T03:47:43.628 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7fbed8001c90 con 0x7fbee810a120 2024-03-12T03:47:43.628 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.623+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7fbed8005c60 con 0x7fbee810a120 2024-03-12T03:47:43.629 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.625+0000 7fbef0384640 1 -- 172.21.15.94:0/3347396902 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbecc005080 con 0x7fbee810a120 2024-03-12T03:47:43.629 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.625+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7fbed801c530 con 0x7fbee810a120 2024-03-12T03:47:43.629 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.625+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(805..805 src has 254..805) v4 ==== 7830+0+0 (unknown 1139330870 0 0) 0x7fbed80897a0 con 0x7fbee810a120 2024-03-12T03:47:43.633 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.630+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7fbed8056b00 con 0x7fbee810a120 2024-03-12T03:47:43.816 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:43.814+0000 7fbef0384640 1 -- 172.21.15.94:0/3347396902 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7fbecc005370 con 0x7fbee810a120 2024-03-12T03:47:43.928 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:43 smithi151 ceph-mon[32923]: pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' no longer out of quota; removing NO_QUOTA flag 2024-03-12T03:47:43.928 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:43 smithi151 ceph-mon[32923]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-12T03:47:43.928 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:43 smithi151 ceph-mon[32923]: osdmap e805: 8 total, 8 up, 8 in 2024-03-12T03:47:43.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:43 smithi094 ceph-mon[29291]: pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' no longer out of quota; removing NO_QUOTA flag 2024-03-12T03:47:43.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:43 smithi094 ceph-mon[29291]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-12T03:47:43.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:43 smithi094 ceph-mon[29291]: osdmap e805: 8 total, 8 up, 8 in 2024-03-12T03:47:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:43 smithi094 ceph-mon[40484]: pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' no longer out of quota; removing NO_QUOTA flag 2024-03-12T03:47:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:43 smithi094 ceph-mon[40484]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-12T03:47:43.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:43 smithi094 ceph-mon[40484]: osdmap e805: 8 total, 8 up, 8 in 2024-03-12T03:47:44.645 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:44.643+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v806) v1 ==== 221+0+0 (unknown 4212932104 0 0) 0x7fbed805a550 con 0x7fbee810a120 2024-03-12T03:47:44.759 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:44.757+0000 7fbef0384640 1 -- 172.21.15.94:0/3347396902 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7fbecc000c40 con 0x7fbee810a120 2024-03-12T03:47:44.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:44 smithi094 ceph-mon[29291]: pgmap v1738: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:44.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:44 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3347396902' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-12T03:47:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:44 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:44.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:44 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:44 smithi094 ceph-mon[40484]: pgmap v1738: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:44 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3347396902' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-12T03:47:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:44 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:44.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:44 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:44 smithi151 ceph-mon[32923]: pgmap v1738: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:44 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3347396902' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-12T03:47:44.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:44 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:44.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:44 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:45.652 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:45.649+0000 7fbedf7fe640 1 -- 172.21.15.94:0/3347396902 <== mon.0 v1:172.21.15.94:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v807) v1 ==== 221+0+0 (unknown 2068639721 0 0) 0x7fbed805eba0 con 0x7fbee810a120 2024-03-12T03:47:45.652 INFO:tasks.workunit.client.0.smithi094.stderr:set-quota max_bytes = 100 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 2024-03-12T03:47:45.656 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:45.654+0000 7fbef0384640 1 -- 172.21.15.94:0/3347396902 >> v1:172.21.15.94:6800/2099808905 conn(0x7fbed406ebd0 legacy=0x7fbed4071090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:45.656 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:45.654+0000 7fbef0384640 1 -- 172.21.15.94:0/3347396902 >> v1:172.21.15.94:6789/0 conn(0x7fbee810a120 legacy=0x7fbee81a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:47:45.656 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:45.654+0000 7fbef0384640 1 -- 172.21.15.94:0/3347396902 shutdown_connections 2024-03-12T03:47:45.657 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:45.654+0000 7fbef0384640 1 -- 172.21.15.94:0/3347396902 >> 172.21.15.94:0/3347396902 conn(0x7fbee80fbb40 msgr2=0x7fbee8101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:47:45.657 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:45.654+0000 7fbef0384640 1 -- 172.21.15.94:0/3347396902 shutdown_connections 2024-03-12T03:47:45.657 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:47:45.654+0000 7fbef0384640 1 -- 172.21.15.94:0/3347396902 wait complete. 2024-03-12T03:47:45.668 INFO:tasks.workunit.client.0.smithi094.stderr:+ sleep 30 2024-03-12T03:47:45.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3347396902' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]': finished 2024-03-12T03:47:45.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:45 smithi094 ceph-mon[29291]: osdmap e806: 8 total, 8 up, 8 in 2024-03-12T03:47:45.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:45 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3347396902' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-12T03:47:45.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:45 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:45.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:45 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3347396902' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]': finished 2024-03-12T03:47:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:45 smithi094 ceph-mon[40484]: osdmap e806: 8 total, 8 up, 8 in 2024-03-12T03:47:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:45 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3347396902' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-12T03:47:45.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:45 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:45.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:45 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3347396902' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]': finished 2024-03-12T03:47:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:45 smithi151 ceph-mon[32923]: osdmap e806: 8 total, 8 up, 8 in 2024-03-12T03:47:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:45 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3347396902' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-12T03:47:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:45 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:45.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:45 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:46.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:46 smithi094 ceph-mon[29291]: pgmap v1741: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:47:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:46 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3347396902' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]': finished 2024-03-12T03:47:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:46 smithi094 ceph-mon[29291]: osdmap e807: 8 total, 8 up, 8 in 2024-03-12T03:47:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:46 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:47:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:46 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:47:46.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:46 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:46 smithi094 ceph-mon[40484]: pgmap v1741: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:47:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:46 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3347396902' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]': finished 2024-03-12T03:47:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:46 smithi094 ceph-mon[40484]: osdmap e807: 8 total, 8 up, 8 in 2024-03-12T03:47:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:46 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:47:46.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:46 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:47:46.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:46 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:46 smithi151 ceph-mon[32923]: pgmap v1741: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:47:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:46 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3347396902' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "100"}]': finished 2024-03-12T03:47:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:46 smithi151 ceph-mon[32923]: osdmap e807: 8 total, 8 up, 8 in 2024-03-12T03:47:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:46 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:47:46.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:46 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:47:46.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:46 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:47:48.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:48 smithi094 ceph-mon[29291]: pgmap v1743: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-03-12T03:47:48.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:48 smithi094 ceph-mon[29291]: pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' is full (reached quota's max_bytes: 100 B) 2024-03-12T03:47:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:48 smithi094 ceph-mon[29291]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-12T03:47:48.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:48 smithi094 ceph-mon[29291]: osdmap e808: 8 total, 8 up, 8 in 2024-03-12T03:47:48.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:48 smithi094 ceph-mon[40484]: pgmap v1743: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-03-12T03:47:48.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:48 smithi094 ceph-mon[40484]: pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' is full (reached quota's max_bytes: 100 B) 2024-03-12T03:47:48.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:48 smithi094 ceph-mon[40484]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-12T03:47:48.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:48 smithi094 ceph-mon[40484]: osdmap e808: 8 total, 8 up, 8 in 2024-03-12T03:47:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:48 smithi151 ceph-mon[32923]: pgmap v1743: 176 pgs: 176 active+clean; 697 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-03-12T03:47:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:48 smithi151 ceph-mon[32923]: pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' is full (reached quota's max_bytes: 100 B) 2024-03-12T03:47:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:48 smithi151 ceph-mon[32923]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-12T03:47:48.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:48 smithi151 ceph-mon[32923]: osdmap e808: 8 total, 8 up, 8 in 2024-03-12T03:47:49.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:47:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:47:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:50 smithi094 ceph-mon[29291]: pgmap v1745: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 895 B/s rd, 4.2 KiB/s wr, 1 op/s 2024-03-12T03:47:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:50.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:50.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:47:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:47:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:47:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:50 smithi094 ceph-mon[40484]: pgmap v1745: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 895 B/s rd, 4.2 KiB/s wr, 1 op/s 2024-03-12T03:47:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:50.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:50 smithi151 ceph-mon[32923]: pgmap v1745: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 895 B/s rd, 4.2 KiB/s wr, 1 op/s 2024-03-12T03:47:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:50.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:52.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:52 smithi094 ceph-mon[29291]: pgmap v1746: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 780 B/s rd, 3.7 KiB/s wr, 1 op/s 2024-03-12T03:47:52.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:52 smithi094 ceph-mon[40484]: pgmap v1746: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 780 B/s rd, 3.7 KiB/s wr, 1 op/s 2024-03-12T03:47:52.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:52 smithi151 ceph-mon[32923]: pgmap v1746: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 780 B/s rd, 3.7 KiB/s wr, 1 op/s 2024-03-12T03:47:54.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:54 smithi094 ceph-mon[40484]: pgmap v1747: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-12T03:47:54.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:54 smithi094 ceph-mon[29291]: pgmap v1747: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-12T03:47:54.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:54 smithi151 ceph-mon[32923]: pgmap v1747: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-12T03:47:56.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:56 smithi094 ceph-mon[29291]: pgmap v1748: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.5 KiB/s wr, 1 op/s 2024-03-12T03:47:56.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:56 smithi094 ceph-mon[40484]: pgmap v1748: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.5 KiB/s wr, 1 op/s 2024-03-12T03:47:56.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:56 smithi151 ceph-mon[32923]: pgmap v1748: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.5 KiB/s wr, 1 op/s 2024-03-12T03:47:58.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:47:58 smithi094 ceph-mon[29291]: pgmap v1749: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-12T03:47:58.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:47:58 smithi094 ceph-mon[40484]: pgmap v1749: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-12T03:47:58.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:47:58 smithi151 ceph-mon[32923]: pgmap v1749: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-12T03:47:59.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:47:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:48:00.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:00 smithi094 ceph-mon[29291]: pgmap v1750: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-12T03:48:00.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:00.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:48:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:48:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:48:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:00 smithi094 ceph-mon[40484]: pgmap v1750: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-12T03:48:00.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:00 smithi151 ceph-mon[32923]: pgmap v1750: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-12T03:48:00.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:02.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:02 smithi094 ceph-mon[29291]: pgmap v1751: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:02.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:02 smithi094 ceph-mon[40484]: pgmap v1751: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:02.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:02 smithi151 ceph-mon[32923]: pgmap v1751: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:04.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:04 smithi094 ceph-mon[29291]: pgmap v1752: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:04.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:04 smithi094 ceph-mon[40484]: pgmap v1752: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:04.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:04 smithi151 ceph-mon[32923]: pgmap v1752: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:05.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:05 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:05.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:05 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:05.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:05 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:06 smithi094 ceph-mon[29291]: pgmap v1753: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:06 smithi094 ceph-mon[40484]: pgmap v1753: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:06 smithi151 ceph-mon[32923]: pgmap v1753: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:08.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:08 smithi094 ceph-mon[29291]: pgmap v1754: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:08.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:08 smithi094 ceph-mon[40484]: pgmap v1754: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:08 smithi151 ceph-mon[32923]: pgmap v1754: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:09.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:48:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:48:10.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:10 smithi094 ceph-mon[29291]: pgmap v1755: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:10 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:48:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:48:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:48:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:10 smithi094 ceph-mon[40484]: pgmap v1755: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:10 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:10 smithi151 ceph-mon[32923]: pgmap v1755: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:10 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:12.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:12 smithi094 ceph-mon[29291]: pgmap v1756: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:12 smithi094 ceph-mon[40484]: pgmap v1756: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:12 smithi151 ceph-mon[32923]: pgmap v1756: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:14.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:14 smithi094 ceph-mon[40484]: pgmap v1757: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:14.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:14 smithi094 ceph-mon[29291]: pgmap v1757: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:14 smithi151 ceph-mon[32923]: pgmap v1757: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:15.670 INFO:tasks.workunit.client.0.smithi094.stderr:+ pid=154922 2024-03-12T03:48:15.670 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool set-quota f98a828b-0804-4d73-9e78-2d8570d1ebe3 max_bytes 0 2024-03-12T03:48:15.670 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put two /etc/passwd 2024-03-12T03:48:15.765 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.760+0000 7f8964790640 1 -- 172.21.15.94:0/476384080 >> v1:172.21.15.94:6789/0 conn(0x7f895c10a010 legacy=0x7f895c10c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:15.765 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.760+0000 7f8964790640 1 -- 172.21.15.94:0/476384080 shutdown_connections 2024-03-12T03:48:15.766 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.760+0000 7f8964790640 1 -- 172.21.15.94:0/476384080 >> 172.21.15.94:0/476384080 conn(0x7f895c0fcbd0 msgr2=0x7f895c0feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:48:15.766 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.760+0000 7f8964790640 1 -- 172.21.15.94:0/476384080 shutdown_connections 2024-03-12T03:48:15.766 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.760+0000 7f8964790640 1 -- 172.21.15.94:0/476384080 wait complete. 2024-03-12T03:48:15.766 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.761+0000 7f8964790640 1 Processor -- start 2024-03-12T03:48:15.766 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.761+0000 7f8964790640 1 -- start start 2024-03-12T03:48:15.766 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.761+0000 7f8964790640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f895c1a5fd0 con 0x7f895c10a010 2024-03-12T03:48:15.766 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.761+0000 7f8964790640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f895c1a7210 con 0x7f895c10db80 2024-03-12T03:48:15.766 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.761+0000 7f8964790640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f895c1a8450 con 0x7f895c101390 2024-03-12T03:48:15.766 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.761+0000 7f896252c640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f895c101390 0x7f895c1064e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:58622/0 (socket says 172.21.15.94:58622) 2024-03-12T03:48:15.766 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.761+0000 7f896252c640 1 -- 172.21.15.94:0/3652722078 learned_addr learned my addr 172.21.15.94:0/3652722078 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:48:15.766 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.761+0000 7f8961d2b640 1 --1- 172.21.15.94:0/3652722078 >> v1:172.21.15.94:6789/0 conn(0x7f895c10a010 0x7f895c1a0ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56666/0 (socket says 172.21.15.94:56666) 2024-03-12T03:48:15.766 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.761+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931033639 0 0) 0x7f895c1a8450 con 0x7f895c101390 2024-03-12T03:48:15.766 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.761+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8948003520 con 0x7f895c101390 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138600041 0 0) 0x7f895c1a5fd0 con 0x7f895c10a010 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f895c1a8450 con 0x7f895c10a010 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274076787 0 0) 0x7f895c1a7210 con 0x7f895c10db80 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f895c1a5fd0 con 0x7f895c10db80 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 28880794 0 0) 0x7f8948003520 con 0x7f895c101390 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f895c1a7210 con 0x7f895c101390 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f894c001a30 con 0x7f895c101390 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 410841111 0 0) 0x7f895c1a8450 con 0x7f895c10a010 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8948003520 con 0x7f895c10a010 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f89580043a0 con 0x7f895c10a010 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 479119201 0 0) 0x7f895c1a5fd0 con 0x7f895c10db80 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f895c1a8450 con 0x7f895c10db80 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2847112223 0 0) 0x7f895c1a7210 con 0x7f895c101390 2024-03-12T03:48:15.767 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 >> v1:172.21.15.151:6789/0 conn(0x7f895c10db80 legacy=0x7f895c1a4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:15.768 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 >> v1:172.21.15.94:6789/0 conn(0x7f895c10a010 legacy=0x7f895c1a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:15.768 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895c1a9690 con 0x7f895c101390 2024-03-12T03:48:15.768 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f8964790640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f895c1a6200 con 0x7f895c101390 2024-03-12T03:48:15.768 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f8964790640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f895c1a66d0 con 0x7f895c101390 2024-03-12T03:48:15.768 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f894c001bd0 con 0x7f895c101390 2024-03-12T03:48:15.768 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.762+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f894c005c40 con 0x7f895c101390 2024-03-12T03:48:15.768 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.763+0000 7f8964790640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8940005080 con 0x7f895c101390 2024-03-12T03:48:15.768 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.764+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f894c003fa0 con 0x7f895c101390 2024-03-12T03:48:15.769 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.764+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(808..808 src has 254..808) v4 ==== 7830+0+0 (unknown 3015928069 0 0) 0x7f894c089580 con 0x7f895c101390 2024-03-12T03:48:15.769 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.764+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=809}) v3 -- 0x7f895c1a7210 con 0x7f895c101390 2024-03-12T03:48:15.771 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.769+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f894c056800 con 0x7f895c101390 2024-03-12T03:48:15.946 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:15.944+0000 7f8964790640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f8940005370 con 0x7f895c101390 2024-03-12T03:48:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:16.696+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.2 v1:172.21.15.94:6790/0 10 ==== osd_map(809..809 src has 254..809) v4 ==== 628+0+0 (unknown 710265577 0 0) 0x7f894c087370 con 0x7f895c101390 2024-03-12T03:48:16.699 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:16.696+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=810}) v3 -- 0x7f8948003520 con 0x7f895c101390 2024-03-12T03:48:16.703 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:16.700+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.2 v1:172.21.15.94:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v809) v1 ==== 217+0+0 (unknown 3769155357 0 0) 0x7f894c05a250 con 0x7f895c101390 2024-03-12T03:48:16.814 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:16.811+0000 7f8964790640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f8940000c40 con 0x7f895c101390 2024-03-12T03:48:16.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:16 smithi094 ceph-mon[29291]: pgmap v1758: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:16.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:16 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3652722078' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:48:16.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:16 smithi094 ceph-mon[29291]: from='client.57314 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:48:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:16 smithi094 ceph-mon[40484]: pgmap v1758: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:16 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3652722078' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:48:16.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:16 smithi094 ceph-mon[40484]: from='client.57314 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:48:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:16 smithi151 ceph-mon[32923]: pgmap v1758: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:16 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3652722078' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:48:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:16 smithi151 ceph-mon[32923]: from='client.57314 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:48:17.712 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.709+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.2 v1:172.21.15.94:6790/0 12 ==== osd_map(810..810 src has 254..810) v4 ==== 628+0+0 (unknown 2554263024 0 0) 0x7f894c0526e0 con 0x7f895c101390 2024-03-12T03:48:17.712 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.709+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=811}) v3 -- 0x7f895c1a5fd0 con 0x7f895c101390 2024-03-12T03:48:17.717 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.714+0000 7f89537fe640 1 -- 172.21.15.94:0/3652722078 <== mon.2 v1:172.21.15.94:6790/0 13 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v810) v1 ==== 217+0+0 (unknown 2527715262 0 0) 0x7f894c05e8a0 con 0x7f895c101390 2024-03-12T03:48:17.717 INFO:tasks.workunit.client.0.smithi094.stderr:set-quota max_bytes = 0 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 2024-03-12T03:48:17.720 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.718+0000 7f8964790640 1 -- 172.21.15.94:0/3652722078 >> v1:172.21.15.94:6800/2099808905 conn(0x7f894806eba0 legacy=0x7f8948071060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:17.721 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.718+0000 7f8964790640 1 -- 172.21.15.94:0/3652722078 >> v1:172.21.15.94:6790/0 conn(0x7f895c101390 legacy=0x7f895c1064e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:17.721 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.718+0000 7f8964790640 1 -- 172.21.15.94:0/3652722078 shutdown_connections 2024-03-12T03:48:17.721 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.718+0000 7f8964790640 1 -- 172.21.15.94:0/3652722078 >> 172.21.15.94:0/3652722078 conn(0x7f895c0fcbd0 msgr2=0x7f895c10e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:48:17.721 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.718+0000 7f8964790640 1 -- 172.21.15.94:0/3652722078 shutdown_connections 2024-03-12T03:48:17.721 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.718+0000 7f8964790640 1 -- 172.21.15.94:0/3652722078 wait complete. 2024-03-12T03:48:17.731 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool set-quota f98a828b-0804-4d73-9e78-2d8570d1ebe3 max_objects 0 2024-03-12T03:48:17.816 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.814+0000 7f505e09a640 1 -- 172.21.15.94:0/3709635983 >> v1:172.21.15.94:6789/0 conn(0x7f5058108f20 legacy=0x7f505810b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:17.816 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.814+0000 7f505e09a640 1 -- 172.21.15.94:0/3709635983 shutdown_connections 2024-03-12T03:48:17.817 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.814+0000 7f505e09a640 1 -- 172.21.15.94:0/3709635983 >> 172.21.15.94:0/3709635983 conn(0x7f5058076d00 msgr2=0x7f50580fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:48:17.817 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.814+0000 7f505e09a640 1 -- 172.21.15.94:0/3709635983 shutdown_connections 2024-03-12T03:48:17.817 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.814+0000 7f505e09a640 1 -- 172.21.15.94:0/3709635983 wait complete. 2024-03-12T03:48:17.819 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.814+0000 7f505e09a640 1 Processor -- start 2024-03-12T03:48:17.819 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f505e09a640 1 -- start start 2024-03-12T03:48:17.819 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f505e09a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f50581a61d0 con 0x7f50581002a0 2024-03-12T03:48:17.819 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f505e09a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f50581a73d0 con 0x7f505810ca90 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f505e09a640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f50581a8610 con 0x7f5058108f20 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f50577fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f50581002a0 0x7f5058068d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56688/0 (socket says 172.21.15.94:56688) 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f50577fe640 1 -- 172.21.15.94:0/297603686 learned_addr learned my addr 172.21.15.94:0/297603686 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f504effd640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f5058108f20 0x7f50581a0f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:58650/0 (socket says 172.21.15.94:58650) 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189558862 0 0) 0x7f50581a61d0 con 0x7f50581002a0 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5044003520 con 0x7f50581002a0 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871094249 0 0) 0x7f50581a8610 con 0x7f5058108f20 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50581a61d0 con 0x7f5058108f20 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3918260136 0 0) 0x7f50581a61d0 con 0x7f5058108f20 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50581a8610 con 0x7f5058108f20 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.815+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f5040004480 con 0x7f5058108f20 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.816+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2504906765 0 0) 0x7f5044003520 con 0x7f50581002a0 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.816+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50581a61d0 con 0x7f50581002a0 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.816+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462354595 0 0) 0x7f50581a73d0 con 0x7f505810ca90 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.816+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5044003520 con 0x7f505810ca90 2024-03-12T03:48:17.820 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.816+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f5048001a30 con 0x7f50581002a0 2024-03-12T03:48:17.821 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.816+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1832144961 0 0) 0x7f50581a8610 con 0x7f5058108f20 2024-03-12T03:48:17.821 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.816+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 >> v1:172.21.15.151:6789/0 conn(0x7f505810ca90 legacy=0x7f50581a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:17.821 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.816+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 >> v1:172.21.15.94:6789/0 conn(0x7f50581002a0 legacy=0x7f5058068d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:17.821 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.816+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50581a9850 con 0x7f5058108f20 2024-03-12T03:48:17.821 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.816+0000 7f505e09a640 1 -- 172.21.15.94:0/297603686 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50581a6400 con 0x7f5058108f20 2024-03-12T03:48:17.821 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.816+0000 7f505e09a640 1 -- 172.21.15.94:0/297603686 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50581a68d0 con 0x7f5058108f20 2024-03-12T03:48:17.821 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.816+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f50400011c0 con 0x7f5058108f20 2024-03-12T03:48:17.821 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.816+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f5040004c10 con 0x7f5058108f20 2024-03-12T03:48:17.821 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.817+0000 7f505e09a640 1 -- 172.21.15.94:0/297603686 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5038005080 con 0x7f5058108f20 2024-03-12T03:48:17.821 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.817+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f5040001ab0 con 0x7f5058108f20 2024-03-12T03:48:17.822 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.818+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(810..810 src has 254..810) v4 ==== 7830+0+0 (unknown 1294666034 0 0) 0x7f5040089820 con 0x7f5058108f20 2024-03-12T03:48:17.822 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.818+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=811}) v3 -- 0x7f50581a8610 con 0x7f5058108f20 2024-03-12T03:48:17.824 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.822+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f5040056aa0 con 0x7f5058108f20 2024-03-12T03:48:18.001 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:17.998+0000 7f505e09a640 1 -- 172.21.15.94:0/297603686 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f5038005370 con 0x7f5058108f20 2024-03-12T03:48:18.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:17 smithi094 ceph-mon[29291]: from='client.57314 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]': finished 2024-03-12T03:48:18.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:17 smithi094 ceph-mon[29291]: osdmap e809: 8 total, 8 up, 8 in 2024-03-12T03:48:18.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:17 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3652722078' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:48:18.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:17 smithi094 ceph-mon[29291]: from='client.57314 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:48:18.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:17 smithi094 ceph-mon[40484]: from='client.57314 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]': finished 2024-03-12T03:48:18.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:17 smithi094 ceph-mon[40484]: osdmap e809: 8 total, 8 up, 8 in 2024-03-12T03:48:18.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:17 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3652722078' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:48:18.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:17 smithi094 ceph-mon[40484]: from='client.57314 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:48:18.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:17 smithi151 ceph-mon[32923]: from='client.57314 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]': finished 2024-03-12T03:48:18.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:17 smithi151 ceph-mon[32923]: osdmap e809: 8 total, 8 up, 8 in 2024-03-12T03:48:18.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:17 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3652722078' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:48:18.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:17 smithi151 ceph-mon[32923]: from='client.57314 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:48:18.495 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:18.492+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.2 v1:172.21.15.94:6790/0 10 ==== osd_map(811..811 src has 254..811) v4 ==== 628+0+0 (unknown 3904755895 0 0) 0x7f5040087610 con 0x7f5058108f20 2024-03-12T03:48:18.495 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:18.492+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=812}) v3 -- 0x7f50581a73d0 con 0x7f5058108f20 2024-03-12T03:48:18.499 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:18.497+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.2 v1:172.21.15.94:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v811) v1 ==== 221+0+0 (unknown 64655789 0 0) 0x7f504005a4f0 con 0x7f5058108f20 2024-03-12T03:48:18.609 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:18.607+0000 7f505e09a640 1 -- 172.21.15.94:0/297603686 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f5038000c40 con 0x7f5058108f20 2024-03-12T03:48:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:18 smithi151 ceph-mon[32923]: pgmap v1760: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:48:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:18 smithi151 ceph-mon[32923]: from='client.57314 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]': finished 2024-03-12T03:48:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:18 smithi151 ceph-mon[32923]: osdmap e810: 8 total, 8 up, 8 in 2024-03-12T03:48:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/297603686' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:48:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:18 smithi151 ceph-mon[32923]: from='client.57320 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:48:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:18 smithi151 ceph-mon[32923]: pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' no longer out of quota; removing NO_QUOTA flag 2024-03-12T03:48:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:18 smithi151 ceph-mon[32923]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-12T03:48:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:18 smithi151 ceph-mon[32923]: from='client.57320 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]': finished 2024-03-12T03:48:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:18 smithi151 ceph-mon[32923]: osdmap e811: 8 total, 8 up, 8 in 2024-03-12T03:48:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:18 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/297603686' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:48:18.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:18 smithi151 ceph-mon[32923]: from='client.57320 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:48:19.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[29291]: pgmap v1760: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:48:19.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[29291]: from='client.57314 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]': finished 2024-03-12T03:48:19.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[29291]: osdmap e810: 8 total, 8 up, 8 in 2024-03-12T03:48:19.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/297603686' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:48:19.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[29291]: from='client.57320 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:48:19.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[29291]: pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' no longer out of quota; removing NO_QUOTA flag 2024-03-12T03:48:19.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[29291]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-12T03:48:19.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[29291]: from='client.57320 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]': finished 2024-03-12T03:48:19.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[29291]: osdmap e811: 8 total, 8 up, 8 in 2024-03-12T03:48:19.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/297603686' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:48:19.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[29291]: from='client.57320 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:48:19.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[40484]: pgmap v1760: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:48:19.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[40484]: from='client.57314 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]': finished 2024-03-12T03:48:19.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[40484]: osdmap e810: 8 total, 8 up, 8 in 2024-03-12T03:48:19.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/297603686' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:48:19.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[40484]: from='client.57320 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:48:19.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[40484]: pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' no longer out of quota; removing NO_QUOTA flag 2024-03-12T03:48:19.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[40484]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-12T03:48:19.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[40484]: from='client.57320 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]': finished 2024-03-12T03:48:19.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[40484]: osdmap e811: 8 total, 8 up, 8 in 2024-03-12T03:48:19.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/297603686' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:48:19.211 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:18 smithi094 ceph-mon[40484]: from='client.57320 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:48:19.496 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.494+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.2 v1:172.21.15.94:6790/0 12 ==== osd_map(812..812 src has 254..812) v4 ==== 628+0+0 (unknown 3755267429 0 0) 0x7f5040052980 con 0x7f5058108f20 2024-03-12T03:48:19.501 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.498+0000 7f50557fa640 1 -- 172.21.15.94:0/297603686 <== mon.2 v1:172.21.15.94:6790/0 13 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v812) v1 ==== 221+0+0 (unknown 2354855705 0 0) 0x7f504005eb40 con 0x7f5058108f20 2024-03-12T03:48:19.501 INFO:tasks.workunit.client.0.smithi094.stderr:set-quota max_objects = 0 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 2024-03-12T03:48:19.504 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.502+0000 7f505e09a640 1 -- 172.21.15.94:0/297603686 >> v1:172.21.15.94:6800/2099808905 conn(0x7f504406eba0 legacy=0x7f5044071060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:19.504 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.502+0000 7f505e09a640 1 -- 172.21.15.94:0/297603686 >> v1:172.21.15.94:6790/0 conn(0x7f5058108f20 legacy=0x7f50581a0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:19.504 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.502+0000 7f505e09a640 1 -- 172.21.15.94:0/297603686 shutdown_connections 2024-03-12T03:48:19.504 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.502+0000 7f505e09a640 1 -- 172.21.15.94:0/297603686 >> 172.21.15.94:0/297603686 conn(0x7f5058076d00 msgr2=0x7f5058107780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:48:19.504 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.502+0000 7f505e09a640 1 -- 172.21.15.94:0/297603686 shutdown_connections 2024-03-12T03:48:19.505 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.502+0000 7f505e09a640 1 -- 172.21.15.94:0/297603686 wait complete. 2024-03-12T03:48:19.514 INFO:tasks.workunit.client.0.smithi094.stderr:+ wait 154922 2024-03-12T03:48:19.514 INFO:tasks.workunit.client.0.smithi094.stderr:+ '[' 0 -ne 0 ']' 2024-03-12T03:48:19.514 INFO:tasks.workunit.client.0.smithi094.stderr:+ true 2024-03-12T03:48:19.515 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put three /etc/passwd 2024-03-12T03:48:19.548 INFO:tasks.workunit.client.0.smithi094.stderr:++ uuidgen 2024-03-12T03:48:19.549 INFO:tasks.workunit.client.0.smithi094.stderr:+ pp=ac83aab0-2794-4807-a391-8efea46d2288 2024-03-12T03:48:19.549 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool create ac83aab0-2794-4807-a391-8efea46d2288 12 2024-03-12T03:48:19.630 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.627+0000 7f129010a640 1 -- 172.21.15.94:0/901755793 >> v1:172.21.15.94:6790/0 conn(0x7f1288102370 legacy=0x7f1288102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:19.631 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.627+0000 7f129010a640 1 -- 172.21.15.94:0/901755793 shutdown_connections 2024-03-12T03:48:19.631 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.627+0000 7f129010a640 1 -- 172.21.15.94:0/901755793 >> 172.21.15.94:0/901755793 conn(0x7f12880fbb10 msgr2=0x7f12880fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:48:19.631 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.627+0000 7f129010a640 1 -- 172.21.15.94:0/901755793 shutdown_connections 2024-03-12T03:48:19.631 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.627+0000 7f129010a640 1 -- 172.21.15.94:0/901755793 wait complete. 2024-03-12T03:48:19.631 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.628+0000 7f129010a640 1 Processor -- start 2024-03-12T03:48:19.631 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.628+0000 7f129010a640 1 -- start start 2024-03-12T03:48:19.631 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.628+0000 7f129010a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f12881a61b0 con 0x7f128810a170 2024-03-12T03:48:19.631 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.628+0000 7f129010a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f12881a73f0 con 0x7f128810dc80 2024-03-12T03:48:19.631 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.628+0000 7f129010a640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f12881a8630 con 0x7f1288102370 2024-03-12T03:48:19.631 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.628+0000 7f128dea6640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f1288102370 0x7f12881099a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:58690/0 (socket says 172.21.15.94:58690) 2024-03-12T03:48:19.631 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.628+0000 7f128dea6640 1 -- 172.21.15.94:0/3769687149 learned_addr learned my addr 172.21.15.94:0/3769687149 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:48:19.635 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.628+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468790477 0 0) 0x7f12881a8630 con 0x7f1288102370 2024-03-12T03:48:19.636 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.628+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1274003520 con 0x7f1288102370 2024-03-12T03:48:19.636 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.628+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672175385 0 0) 0x7f12881a61b0 con 0x7f128810a170 2024-03-12T03:48:19.636 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.628+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12881a8630 con 0x7f128810a170 2024-03-12T03:48:19.636 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33356940 0 0) 0x7f12881a73f0 con 0x7f128810dc80 2024-03-12T03:48:19.636 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12881a61b0 con 0x7f128810dc80 2024-03-12T03:48:19.636 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1922769174 0 0) 0x7f1274003520 con 0x7f1288102370 2024-03-12T03:48:19.636 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12881a73f0 con 0x7f1288102370 2024-03-12T03:48:19.636 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1284004480 con 0x7f1288102370 2024-03-12T03:48:19.636 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763700887 0 0) 0x7f12881a8630 con 0x7f128810a170 2024-03-12T03:48:19.636 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1274003520 con 0x7f128810a170 2024-03-12T03:48:19.636 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1278001810 con 0x7f128810a170 2024-03-12T03:48:19.637 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2262829804 0 0) 0x7f12881a73f0 con 0x7f1288102370 2024-03-12T03:48:19.637 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 >> v1:172.21.15.151:6789/0 conn(0x7f128810dc80 legacy=0x7f12881a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:19.637 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 >> v1:172.21.15.94:6789/0 conn(0x7f128810a170 legacy=0x7f12881a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:19.637 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12881a9870 con 0x7f1288102370 2024-03-12T03:48:19.637 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f129010a640 1 -- 172.21.15.94:0/3769687149 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12881a63e0 con 0x7f1288102370 2024-03-12T03:48:19.637 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f129010a640 1 -- 172.21.15.94:0/3769687149 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12881a68b0 con 0x7f1288102370 2024-03-12T03:48:19.637 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f1284001ab0 con 0x7f1288102370 2024-03-12T03:48:19.637 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.629+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1284004c90 con 0x7f1288102370 2024-03-12T03:48:19.637 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.630+0000 7f129010a640 1 -- 172.21.15.94:0/3769687149 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126c005080 con 0x7f1288102370 2024-03-12T03:48:19.637 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.630+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f1284004f10 con 0x7f1288102370 2024-03-12T03:48:19.637 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.631+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(812..812 src has 254..812) v4 ==== 7830+0+0 (unknown 1275060122 0 0) 0x7f12840897b0 con 0x7f1288102370 2024-03-12T03:48:19.638 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.635+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f12840569d0 con 0x7f1288102370 2024-03-12T03:48:19.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:48:19 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:48:19.819 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:19.816+0000 7f129010a640 1 -- 172.21.15.94:0/3769687149 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12} v 0) v1 -- 0x7f126c005370 con 0x7f1288102370 2024-03-12T03:48:20.718 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.716+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]=0 pool 'ac83aab0-2794-4807-a391-8efea46d2288' created v813) v1 ==== 176+0+0 (unknown 2635107859 0 0) 0x7f128405a420 con 0x7f1288102370 2024-03-12T03:48:20.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.830+0000 7f129010a640 1 -- 172.21.15.94:0/3769687149 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12} v 0) v1 -- 0x7f126c001170 con 0x7f1288102370 2024-03-12T03:48:20.833 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.830+0000 7f127effd640 1 -- 172.21.15.94:0/3769687149 <== mon.2 v1:172.21.15.94:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]=0 pool 'ac83aab0-2794-4807-a391-8efea46d2288' already exists v813) v1 ==== 183+0+0 (unknown 1114043028 0 0) 0x7f126c001170 con 0x7f1288102370 2024-03-12T03:48:20.833 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'ac83aab0-2794-4807-a391-8efea46d2288' already exists 2024-03-12T03:48:20.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.833+0000 7f129010a640 1 -- 172.21.15.94:0/3769687149 >> v1:172.21.15.94:6800/2099808905 conn(0x7f127406e9a0 legacy=0x7f1274070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:20.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.833+0000 7f129010a640 1 -- 172.21.15.94:0/3769687149 >> v1:172.21.15.94:6790/0 conn(0x7f1288102370 legacy=0x7f12881099a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:20.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.833+0000 7f129010a640 1 -- 172.21.15.94:0/3769687149 shutdown_connections 2024-03-12T03:48:20.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.833+0000 7f129010a640 1 -- 172.21.15.94:0/3769687149 >> 172.21.15.94:0/3769687149 conn(0x7f12880fbb10 msgr2=0x7f12880fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:48:20.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.833+0000 7f129010a640 1 -- 172.21.15.94:0/3769687149 shutdown_connections 2024-03-12T03:48:20.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.833+0000 7f129010a640 1 -- 172.21.15.94:0/3769687149 wait complete. 2024-03-12T03:48:20.844 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool application enable ac83aab0-2794-4807-a391-8efea46d2288 rados 2024-03-12T03:48:20.929 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.924+0000 7f4f6e468640 1 -- 172.21.15.94:0/3784116403 >> v1:172.21.15.94:6790/0 conn(0x7f4f6810a030 legacy=0x7f4f6810c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.924+0000 7f4f6e468640 1 -- 172.21.15.94:0/3784116403 shutdown_connections 2024-03-12T03:48:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.924+0000 7f4f6e468640 1 -- 172.21.15.94:0/3784116403 >> 172.21.15.94:0/3784116403 conn(0x7f4f680fcc30 msgr2=0x7f4f680ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:48:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.924+0000 7f4f6e468640 1 -- 172.21.15.94:0/3784116403 shutdown_connections 2024-03-12T03:48:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.924+0000 7f4f6e468640 1 -- 172.21.15.94:0/3784116403 wait complete. 2024-03-12T03:48:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.924+0000 7f4f6e468640 1 Processor -- start 2024-03-12T03:48:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.924+0000 7f4f6e468640 1 -- start start 2024-03-12T03:48:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f6e468640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4f68106ae0 con 0x7f4f6810dba0 2024-03-12T03:48:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f6e468640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4f681a7770 con 0x7f4f681013b0 2024-03-12T03:48:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f6e468640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4f681a8950 con 0x7f4f6810a030 2024-03-12T03:48:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f677fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f4f6810a030 0x7f4f68104070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:58718/0 (socket says 172.21.15.94:58718) 2024-03-12T03:48:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f6ca05640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4f6810dba0 0x7f4f681a5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56756/0 (socket says 172.21.15.94:56756) 2024-03-12T03:48:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f677fe640 1 -- 172.21.15.94:0/1862904749 learned_addr learned my addr 172.21.15.94:0/1862904749 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:48:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331748993 0 0) 0x7f4f68106ae0 con 0x7f4f6810dba0 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f50003520 con 0x7f4f6810dba0 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371124568 0 0) 0x7f4f681a8950 con 0x7f4f6810a030 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f68106ae0 con 0x7f4f6810a030 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571554706 0 0) 0x7f4f681a7770 con 0x7f4f681013b0 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f681a8950 con 0x7f4f681013b0 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2146944062 0 0) 0x7f4f68106ae0 con 0x7f4f6810a030 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f681a7770 con 0x7f4f6810a030 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 208379419 0 0) 0x7f4f50003520 con 0x7f4f6810dba0 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f68106ae0 con 0x7f4f6810dba0 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f4f5c004430 con 0x7f4f6810a030 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f4f60001bd0 con 0x7f4f6810dba0 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4242641814 0 0) 0x7f4f681a8950 con 0x7f4f681013b0 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f50003520 con 0x7f4f681013b0 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f4f58001830 con 0x7f4f681013b0 2024-03-12T03:48:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2450312210 0 0) 0x7f4f681a7770 con 0x7f4f6810a030 2024-03-12T03:48:20.932 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 >> v1:172.21.15.151:6789/0 conn(0x7f4f681013b0 legacy=0x7f4f68103960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:20.932 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 >> v1:172.21.15.94:6789/0 conn(0x7f4f6810dba0 legacy=0x7f4f681a5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:20.932 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.925+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f681a9b30 con 0x7f4f6810a030 2024-03-12T03:48:20.932 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.926+0000 7f4f6e468640 1 -- 172.21.15.94:0/1862904749 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f681a8b80 con 0x7f4f6810a030 2024-03-12T03:48:20.932 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.926+0000 7f4f6e468640 1 -- 172.21.15.94:0/1862904749 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f681a9050 con 0x7f4f6810a030 2024-03-12T03:48:20.932 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.926+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f4f5c004710 con 0x7f4f6810a030 2024-03-12T03:48:20.932 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.926+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f4f5c004cf0 con 0x7f4f6810a030 2024-03-12T03:48:20.932 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.927+0000 7f4f6e468640 1 -- 172.21.15.94:0/1862904749 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f54005080 con 0x7f4f6810a030 2024-03-12T03:48:20.932 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.927+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f4f5c001b30 con 0x7f4f6810a030 2024-03-12T03:48:20.932 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.927+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(813..813 src has 254..813) v4 ==== 8205+0+0 (unknown 627879197 0 0) 0x7f4f5c089940 con 0x7f4f6810a030 2024-03-12T03:48:20.934 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:20.932+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f4f5c0569e0 con 0x7f4f6810a030 2024-03-12T03:48:20.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:48:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:48:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:48:20.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:20 smithi094 ceph-mon[29291]: pgmap v1763: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:20.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:20.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:20 smithi094 ceph-mon[29291]: from='client.57320 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]': finished 2024-03-12T03:48:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:20 smithi094 ceph-mon[29291]: osdmap e812: 8 total, 8 up, 8 in 2024-03-12T03:48:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:20 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3769687149' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]: dispatch 2024-03-12T03:48:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:20 smithi094 ceph-mon[29291]: from='client.57335 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]: dispatch 2024-03-12T03:48:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:20 smithi094 ceph-mon[40484]: pgmap v1763: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:20 smithi094 ceph-mon[40484]: from='client.57320 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]': finished 2024-03-12T03:48:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:20 smithi094 ceph-mon[40484]: osdmap e812: 8 total, 8 up, 8 in 2024-03-12T03:48:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:20 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3769687149' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]: dispatch 2024-03-12T03:48:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:20 smithi094 ceph-mon[40484]: from='client.57335 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]: dispatch 2024-03-12T03:48:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:20 smithi151 ceph-mon[32923]: pgmap v1763: 176 pgs: 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:20 smithi151 ceph-mon[32923]: from='client.57320 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]': finished 2024-03-12T03:48:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:20 smithi151 ceph-mon[32923]: osdmap e812: 8 total, 8 up, 8 in 2024-03-12T03:48:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:20 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3769687149' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]: dispatch 2024-03-12T03:48:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:20 smithi151 ceph-mon[32923]: from='client.57335 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]: dispatch 2024-03-12T03:48:21.110 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:21.108+0000 7f4f6e468640 1 -- 172.21.15.94:0/1862904749 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"} v 0) v1 -- 0x7f4f54005370 con 0x7f4f6810a030 2024-03-12T03:48:21.724 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:21.722+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]=0 enabled application 'rados' on pool 'ac83aab0-2794-4807-a391-8efea46d2288' v814) v1 ==== 213+0+0 (unknown 1302857991 0 0) 0x7f4f5c05a430 con 0x7f4f6810a030 2024-03-12T03:48:21.838 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:21.836+0000 7f4f6e468640 1 -- 172.21.15.94:0/1862904749 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"} v 0) v1 -- 0x7f4f54001090 con 0x7f4f6810a030 2024-03-12T03:48:21.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:48:21.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[29291]: from='client.57335 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]': finished 2024-03-12T03:48:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[29291]: osdmap e813: 8 total, 8 up, 8 in 2024-03-12T03:48:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3769687149' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]: dispatch 2024-03-12T03:48:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[29291]: from='client.57335 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]: dispatch 2024-03-12T03:48:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1862904749' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]: dispatch 2024-03-12T03:48:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[29291]: from='client.57341 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]: dispatch 2024-03-12T03:48:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:48:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[40484]: from='client.57335 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]': finished 2024-03-12T03:48:21.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[40484]: osdmap e813: 8 total, 8 up, 8 in 2024-03-12T03:48:21.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3769687149' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]: dispatch 2024-03-12T03:48:21.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[40484]: from='client.57335 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]: dispatch 2024-03-12T03:48:21.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1862904749' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]: dispatch 2024-03-12T03:48:21.961 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:21 smithi094 ceph-mon[40484]: from='client.57341 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]: dispatch 2024-03-12T03:48:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:21 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:48:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:21 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:21 smithi151 ceph-mon[32923]: from='client.57335 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]': finished 2024-03-12T03:48:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:21 smithi151 ceph-mon[32923]: osdmap e813: 8 total, 8 up, 8 in 2024-03-12T03:48:21.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3769687149' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]: dispatch 2024-03-12T03:48:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:21 smithi151 ceph-mon[32923]: from='client.57335 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pg_num": 12}]: dispatch 2024-03-12T03:48:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:21 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1862904749' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]: dispatch 2024-03-12T03:48:21.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:21 smithi151 ceph-mon[32923]: from='client.57341 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]: dispatch 2024-03-12T03:48:22.726 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.724+0000 7f4f657fa640 1 -- 172.21.15.94:0/1862904749 <== mon.2 v1:172.21.15.94:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]=0 enabled application 'rados' on pool 'ac83aab0-2794-4807-a391-8efea46d2288' v815) v1 ==== 213+0+0 (unknown 2005532641 0 0) 0x7f4f5c05ea80 con 0x7f4f6810a030 2024-03-12T03:48:22.727 INFO:tasks.workunit.client.0.smithi094.stderr:enabled application 'rados' on pool 'ac83aab0-2794-4807-a391-8efea46d2288' 2024-03-12T03:48:22.730 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.727+0000 7f4f6e468640 1 -- 172.21.15.94:0/1862904749 >> v1:172.21.15.94:6800/2099808905 conn(0x7f4f5006ec90 legacy=0x7f4f50071150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:22.730 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.727+0000 7f4f6e468640 1 -- 172.21.15.94:0/1862904749 >> v1:172.21.15.94:6790/0 conn(0x7f4f6810a030 legacy=0x7f4f68104070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:22.730 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.727+0000 7f4f6e468640 1 -- 172.21.15.94:0/1862904749 shutdown_connections 2024-03-12T03:48:22.730 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.727+0000 7f4f6e468640 1 -- 172.21.15.94:0/1862904749 >> 172.21.15.94:0/1862904749 conn(0x7f4f680fcc30 msgr2=0x7f4f68110810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:48:22.730 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.727+0000 7f4f6e468640 1 -- 172.21.15.94:0/1862904749 shutdown_connections 2024-03-12T03:48:22.730 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.727+0000 7f4f6e468640 1 -- 172.21.15.94:0/1862904749 wait complete. 2024-03-12T03:48:22.740 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool set-quota ac83aab0-2794-4807-a391-8efea46d2288 max_objects 10 2024-03-12T03:48:22.830 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.825+0000 7f30055fb640 1 -- 172.21.15.94:0/1493015515 >> v1:172.21.15.94:6789/0 conn(0x7f30001002b0 legacy=0x7f3000100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:22.830 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.825+0000 7f30055fb640 1 -- 172.21.15.94:0/1493015515 shutdown_connections 2024-03-12T03:48:22.830 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.825+0000 7f30055fb640 1 -- 172.21.15.94:0/1493015515 >> 172.21.15.94:0/1493015515 conn(0x7f30000fbb10 msgr2=0x7f30000fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:48:22.830 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.825+0000 7f30055fb640 1 -- 172.21.15.94:0/1493015515 shutdown_connections 2024-03-12T03:48:22.830 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.825+0000 7f30055fb640 1 -- 172.21.15.94:0/1493015515 wait complete. 2024-03-12T03:48:22.830 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.825+0000 7f30055fb640 1 Processor -- start 2024-03-12T03:48:22.830 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f30055fb640 1 -- start start 2024-03-12T03:48:22.831 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f30055fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f30001a6230 con 0x7f3000108f30 2024-03-12T03:48:22.831 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f30055fb640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f30001a7470 con 0x7f30001002b0 2024-03-12T03:48:22.831 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f30055fb640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f30001a86b0 con 0x7f300010caa0 2024-03-12T03:48:22.831 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2ffe7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3000108f30 0x7f30001a0f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56784/0 (socket says 172.21.15.94:56784) 2024-03-12T03:48:22.831 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2ffe7fc640 1 -- 172.21.15.94:0/3740391220 learned_addr learned my addr 172.21.15.94:0/3740391220 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:48:22.831 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273935410 0 0) 0x7f30001a6230 con 0x7f3000108f30 2024-03-12T03:48:22.831 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fe8003160 con 0x7f3000108f30 2024-03-12T03:48:22.831 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584838052 0 0) 0x7f30001a86b0 con 0x7f300010caa0 2024-03-12T03:48:22.831 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30001a6230 con 0x7f300010caa0 2024-03-12T03:48:22.831 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058792695 0 0) 0x7f30001a7470 con 0x7f30001002b0 2024-03-12T03:48:22.831 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30001a86b0 con 0x7f30001002b0 2024-03-12T03:48:22.831 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3532109605 0 0) 0x7f2fe8003160 con 0x7f3000108f30 2024-03-12T03:48:22.831 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30001a7470 con 0x7f3000108f30 2024-03-12T03:48:22.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f2ff4001930 con 0x7f3000108f30 2024-03-12T03:48:22.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1153663404 0 0) 0x7f30001a6230 con 0x7f300010caa0 2024-03-12T03:48:22.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fe8003160 con 0x7f300010caa0 2024-03-12T03:48:22.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f2ff80013f0 con 0x7f300010caa0 2024-03-12T03:48:22.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 88437691 0 0) 0x7f30001a7470 con 0x7f3000108f30 2024-03-12T03:48:22.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 >> v1:172.21.15.94:6790/0 conn(0x7f300010caa0 legacy=0x7f30001a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:22.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 >> v1:172.21.15.151:6789/0 conn(0x7f30001002b0 legacy=0x7f3000105670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:22.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.826+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30001a98f0 con 0x7f3000108f30 2024-03-12T03:48:22.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.827+0000 7f30055fb640 1 -- 172.21.15.94:0/3740391220 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30001a88e0 con 0x7f3000108f30 2024-03-12T03:48:22.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.827+0000 7f30055fb640 1 -- 172.21.15.94:0/3740391220 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30001a8db0 con 0x7f3000108f30 2024-03-12T03:48:22.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.827+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f2ff4001d00 con 0x7f3000108f30 2024-03-12T03:48:22.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.827+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f2ff4005d10 con 0x7f3000108f30 2024-03-12T03:48:22.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.828+0000 7f30055fb640 1 -- 172.21.15.94:0/3740391220 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec005080 con 0x7f3000108f30 2024-03-12T03:48:22.833 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.828+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f2ff40038c0 con 0x7f3000108f30 2024-03-12T03:48:22.833 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.829+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(815..815 src has 254..815) v4 ==== 8218+0+0 (unknown 3084478460 0 0) 0x7f2ff4089b90 con 0x7f3000108f30 2024-03-12T03:48:22.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:22.833+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f2ff4056c20 con 0x7f3000108f30 2024-03-12T03:48:22.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:22 smithi094 ceph-mon[40484]: pgmap v1766: 188 pgs: 12 unknown, 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:22.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:22 smithi094 ceph-mon[40484]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:48:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:22 smithi094 ceph-mon[40484]: from='client.57341 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]': finished 2024-03-12T03:48:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:22 smithi094 ceph-mon[40484]: osdmap e814: 8 total, 8 up, 8 in 2024-03-12T03:48:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:22 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1862904749' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]: dispatch 2024-03-12T03:48:22.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:22 smithi094 ceph-mon[40484]: from='client.57341 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]: dispatch 2024-03-12T03:48:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:22 smithi094 ceph-mon[29291]: pgmap v1766: 188 pgs: 12 unknown, 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:22 smithi094 ceph-mon[29291]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:48:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:22 smithi094 ceph-mon[29291]: from='client.57341 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]': finished 2024-03-12T03:48:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:22 smithi094 ceph-mon[29291]: osdmap e814: 8 total, 8 up, 8 in 2024-03-12T03:48:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:22 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1862904749' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]: dispatch 2024-03-12T03:48:22.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:22 smithi094 ceph-mon[29291]: from='client.57341 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]: dispatch 2024-03-12T03:48:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:22 smithi151 ceph-mon[32923]: pgmap v1766: 188 pgs: 12 unknown, 176 active+clean; 720 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:22 smithi151 ceph-mon[32923]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:48:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:22 smithi151 ceph-mon[32923]: from='client.57341 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]': finished 2024-03-12T03:48:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:22 smithi151 ceph-mon[32923]: osdmap e814: 8 total, 8 up, 8 in 2024-03-12T03:48:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:22 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1862904749' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]: dispatch 2024-03-12T03:48:22.965 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:22 smithi151 ceph-mon[32923]: from='client.57341 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]: dispatch 2024-03-12T03:48:23.011 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:23.009+0000 7f30055fb640 1 -- 172.21.15.94:0/3740391220 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f2fec005370 con 0x7f3000108f30 2024-03-12T03:48:23.727 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:23.724+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool ac83aab0-2794-4807-a391-8efea46d2288 v816) v1 ==== 223+0+0 (unknown 1047584554 0 0) 0x7f2ff405a670 con 0x7f3000108f30 2024-03-12T03:48:23.838 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:23.836+0000 7f30055fb640 1 -- 172.21.15.94:0/3740391220 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f2fec001090 con 0x7f3000108f30 2024-03-12T03:48:24.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:23 smithi094 ceph-mon[29291]: from='client.57341 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]': finished 2024-03-12T03:48:24.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:23 smithi094 ceph-mon[29291]: osdmap e815: 8 total, 8 up, 8 in 2024-03-12T03:48:24.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:23 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3740391220' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:48:24.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:23 smithi094 ceph-mon[40484]: from='client.57341 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]': finished 2024-03-12T03:48:24.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:23 smithi094 ceph-mon[40484]: osdmap e815: 8 total, 8 up, 8 in 2024-03-12T03:48:24.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:23 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3740391220' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:48:24.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:23 smithi151 ceph-mon[32923]: from='client.57341 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "app": "rados"}]': finished 2024-03-12T03:48:24.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:23 smithi151 ceph-mon[32923]: osdmap e815: 8 total, 8 up, 8 in 2024-03-12T03:48:24.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:23 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3740391220' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:48:24.733 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:24.731+0000 7f2fe7fff640 1 -- 172.21.15.94:0/3740391220 <== mon.0 v1:172.21.15.94:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool ac83aab0-2794-4807-a391-8efea46d2288 v817) v1 ==== 223+0+0 (unknown 1999993434 0 0) 0x7f2ff405ecc0 con 0x7f3000108f30 2024-03-12T03:48:24.733 INFO:tasks.workunit.client.0.smithi094.stderr:set-quota max_objects = 10 for pool ac83aab0-2794-4807-a391-8efea46d2288 2024-03-12T03:48:24.737 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:24.735+0000 7f30055fb640 1 -- 172.21.15.94:0/3740391220 >> v1:172.21.15.94:6800/2099808905 conn(0x7f2fe806eb80 legacy=0x7f2fe8071040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:24.737 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:24.735+0000 7f30055fb640 1 -- 172.21.15.94:0/3740391220 >> v1:172.21.15.94:6789/0 conn(0x7f3000108f30 legacy=0x7f30001a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:48:24.737 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:24.735+0000 7f30055fb640 1 -- 172.21.15.94:0/3740391220 shutdown_connections 2024-03-12T03:48:24.737 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:24.735+0000 7f30055fb640 1 -- 172.21.15.94:0/3740391220 >> 172.21.15.94:0/3740391220 conn(0x7f30000fbb10 msgr2=0x7f300010ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:48:24.737 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:24.736+0000 7f30055fb640 1 -- 172.21.15.94:0/3740391220 shutdown_connections 2024-03-12T03:48:24.737 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:48:24.736+0000 7f30055fb640 1 -- 172.21.15.94:0/3740391220 wait complete. 2024-03-12T03:48:24.750 INFO:tasks.workunit.client.0.smithi094.stderr:+ sleep 30 2024-03-12T03:48:25.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:24 smithi094 ceph-mon[29291]: pgmap v1769: 188 pgs: 188 active+clean; 743 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-12T03:48:25.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:24 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3740391220' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]': finished 2024-03-12T03:48:25.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:24 smithi094 ceph-mon[29291]: osdmap e816: 8 total, 8 up, 8 in 2024-03-12T03:48:25.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:24 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3740391220' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:48:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:24 smithi094 ceph-mon[40484]: pgmap v1769: 188 pgs: 188 active+clean; 743 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-12T03:48:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:24 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3740391220' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]': finished 2024-03-12T03:48:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:24 smithi094 ceph-mon[40484]: osdmap e816: 8 total, 8 up, 8 in 2024-03-12T03:48:25.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:24 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3740391220' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:48:25.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:24 smithi151 ceph-mon[32923]: pgmap v1769: 188 pgs: 188 active+clean; 743 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-12T03:48:25.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:24 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3740391220' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]': finished 2024-03-12T03:48:25.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:24 smithi151 ceph-mon[32923]: osdmap e816: 8 total, 8 up, 8 in 2024-03-12T03:48:25.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:24 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3740391220' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]: dispatch 2024-03-12T03:48:26.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3740391220' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]': finished 2024-03-12T03:48:26.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:25 smithi094 ceph-mon[29291]: osdmap e817: 8 total, 8 up, 8 in 2024-03-12T03:48:26.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3740391220' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]': finished 2024-03-12T03:48:26.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:25 smithi094 ceph-mon[40484]: osdmap e817: 8 total, 8 up, 8 in 2024-03-12T03:48:26.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3740391220' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "field": "max_objects", "val": "10"}]': finished 2024-03-12T03:48:26.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:25 smithi151 ceph-mon[32923]: osdmap e817: 8 total, 8 up, 8 in 2024-03-12T03:48:27.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:26 smithi094 ceph-mon[29291]: pgmap v1772: 188 pgs: 188 active+clean; 743 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-12T03:48:27.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:26 smithi094 ceph-mon[40484]: pgmap v1772: 188 pgs: 188 active+clean; 743 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-12T03:48:27.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:26 smithi151 ceph-mon[32923]: pgmap v1772: 188 pgs: 188 active+clean; 743 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-12T03:48:29.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:28 smithi094 ceph-mon[29291]: pgmap v1773: 188 pgs: 188 active+clean; 743 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 932 B/s rd, 4.4 KiB/s wr, 1 op/s 2024-03-12T03:48:29.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:28 smithi094 ceph-mon[29291]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:48:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:28 smithi094 ceph-mon[40484]: pgmap v1773: 188 pgs: 188 active+clean; 743 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 932 B/s rd, 4.4 KiB/s wr, 1 op/s 2024-03-12T03:48:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:28 smithi094 ceph-mon[40484]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:48:29.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:28 smithi151 ceph-mon[32923]: pgmap v1773: 188 pgs: 188 active+clean; 743 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 932 B/s rd, 4.4 KiB/s wr, 1 op/s 2024-03-12T03:48:29.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:28 smithi151 ceph-mon[32923]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:48:29.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:48:29 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:48:30.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:30 smithi094 ceph-mon[29291]: pgmap v1774: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-03-12T03:48:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:30.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:48:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:48:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:48:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:30 smithi094 ceph-mon[40484]: pgmap v1774: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-03-12T03:48:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:31.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:30 smithi151 ceph-mon[32923]: pgmap v1774: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-03-12T03:48:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:33.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:32 smithi094 ceph-mon[40484]: pgmap v1775: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:48:33.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:32 smithi094 ceph-mon[29291]: pgmap v1775: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:48:33.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:32 smithi151 ceph-mon[32923]: pgmap v1775: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:48:35.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:34 smithi094 ceph-mon[29291]: pgmap v1776: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:48:35.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:34 smithi094 ceph-mon[40484]: pgmap v1776: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:48:35.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:34 smithi151 ceph-mon[32923]: pgmap v1776: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:48:36.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:37.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:36 smithi094 ceph-mon[29291]: pgmap v1777: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-03-12T03:48:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:36 smithi094 ceph-mon[40484]: pgmap v1777: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-03-12T03:48:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:36 smithi151 ceph-mon[32923]: pgmap v1777: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-03-12T03:48:39.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:38 smithi094 ceph-mon[29291]: pgmap v1778: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:38 smithi094 ceph-mon[40484]: pgmap v1778: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:39.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:38 smithi151 ceph-mon[32923]: pgmap v1778: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:39.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:48:39 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:48:40.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:40 smithi094 ceph-mon[29291]: pgmap v1779: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:48:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:48:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:48:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:40 smithi094 ceph-mon[40484]: pgmap v1779: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:40 smithi151 ceph-mon[32923]: pgmap v1779: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:41.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:43.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:42 smithi094 ceph-mon[29291]: pgmap v1780: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:48:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:42 smithi094 ceph-mon[40484]: pgmap v1780: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:48:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:42 smithi151 ceph-mon[32923]: pgmap v1780: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:48:45.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:44 smithi094 ceph-mon[29291]: pgmap v1781: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:45.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:44 smithi094 ceph-mon[40484]: pgmap v1781: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:45.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:44 smithi151 ceph-mon[32923]: pgmap v1781: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:47.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:46 smithi094 ceph-mon[29291]: pgmap v1782: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:48:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:46 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:48:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:46 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:48:47.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:46 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:48:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:46 smithi094 ceph-mon[40484]: pgmap v1782: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:48:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:46 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:48:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:46 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:48:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:46 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:48:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:46 smithi151 ceph-mon[32923]: pgmap v1782: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:48:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:46 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:48:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:46 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:48:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:46 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:48:48.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:47 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:48:48.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:47 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:48:48.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:47 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:48:49.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:48 smithi094 ceph-mon[29291]: pgmap v1783: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:48 smithi094 ceph-mon[40484]: pgmap v1783: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:48 smithi151 ceph-mon[32923]: pgmap v1783: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:49.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:48:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:48:50.782 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:48:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:48:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:48:51.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:50 smithi094 ceph-mon[29291]: pgmap v1784: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:51.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:51.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:50 smithi094 ceph-mon[40484]: pgmap v1784: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:50 smithi151 ceph-mon[32923]: pgmap v1784: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:53.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:52 smithi094 ceph-mon[29291]: pgmap v1785: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:53.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:52 smithi094 ceph-mon[40484]: pgmap v1785: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:53.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:52 smithi151 ceph-mon[32923]: pgmap v1785: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:54.751 INFO:tasks.workunit.client.0.smithi094.stderr:++ seq 1 10 2024-03-12T03:48:54.752 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:48:54.752 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p ac83aab0-2794-4807-a391-8efea46d2288 put obj1 /etc/passwd 2024-03-12T03:48:54.791 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:48:54.791 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p ac83aab0-2794-4807-a391-8efea46d2288 put obj2 /etc/passwd 2024-03-12T03:48:54.826 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:48:54.827 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p ac83aab0-2794-4807-a391-8efea46d2288 put obj3 /etc/passwd 2024-03-12T03:48:54.860 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:48:54.860 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p ac83aab0-2794-4807-a391-8efea46d2288 put obj4 /etc/passwd 2024-03-12T03:48:54.894 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:48:54.894 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p ac83aab0-2794-4807-a391-8efea46d2288 put obj5 /etc/passwd 2024-03-12T03:48:54.926 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:48:54.926 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p ac83aab0-2794-4807-a391-8efea46d2288 put obj6 /etc/passwd 2024-03-12T03:48:54.958 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:48:54.958 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p ac83aab0-2794-4807-a391-8efea46d2288 put obj7 /etc/passwd 2024-03-12T03:48:54.990 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:48:54.990 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p ac83aab0-2794-4807-a391-8efea46d2288 put obj8 /etc/passwd 2024-03-12T03:48:55.023 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:48:55.024 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p ac83aab0-2794-4807-a391-8efea46d2288 put obj9 /etc/passwd 2024-03-12T03:48:55.056 INFO:tasks.workunit.client.0.smithi094.stderr:+ for f in `seq 1 10` 2024-03-12T03:48:55.056 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p ac83aab0-2794-4807-a391-8efea46d2288 put obj10 /etc/passwd 2024-03-12T03:48:55.089 INFO:tasks.workunit.client.0.smithi094.stderr:+ sleep 30 2024-03-12T03:48:55.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:54 smithi094 ceph-mon[29291]: pgmap v1786: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:55.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:54 smithi094 ceph-mon[40484]: pgmap v1786: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:55.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:54 smithi151 ceph-mon[32923]: pgmap v1786: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:57.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:56 smithi094 ceph-mon[29291]: pgmap v1787: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:48:57.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:56 smithi094 ceph-mon[40484]: pgmap v1787: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:48:57.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:56 smithi151 ceph-mon[32923]: pgmap v1787: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:48:59.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:48:58 smithi094 ceph-mon[29291]: pgmap v1788: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:59.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:48:58 smithi094 ceph-mon[40484]: pgmap v1788: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:59.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:48:58 smithi151 ceph-mon[32923]: pgmap v1788: 188 pgs: 188 active+clean; 743 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:59.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:48:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:49:00.796 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:49:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:49:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:49:01.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:00 smithi094 ceph-mon[29291]: pgmap v1789: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-12T03:49:01.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:01.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:00 smithi094 ceph-mon[40484]: pgmap v1789: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-12T03:49:01.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:01.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:00 smithi151 ceph-mon[32923]: pgmap v1789: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-12T03:49:01.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:03.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:02 smithi094 ceph-mon[29291]: pgmap v1790: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-12T03:49:03.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:02 smithi094 ceph-mon[40484]: pgmap v1790: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-12T03:49:03.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:02 smithi151 ceph-mon[32923]: pgmap v1790: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-12T03:49:04.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:03 smithi094 ceph-mon[29291]: pool 'ac83aab0-2794-4807-a391-8efea46d2288' is full (reached quota's max_objects: 10) 2024-03-12T03:49:04.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:03 smithi094 ceph-mon[29291]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-12T03:49:04.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:03 smithi094 ceph-mon[29291]: osdmap e818: 8 total, 8 up, 8 in 2024-03-12T03:49:04.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:03 smithi094 ceph-mon[40484]: pool 'ac83aab0-2794-4807-a391-8efea46d2288' is full (reached quota's max_objects: 10) 2024-03-12T03:49:04.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:03 smithi094 ceph-mon[40484]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-12T03:49:04.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:03 smithi094 ceph-mon[40484]: osdmap e818: 8 total, 8 up, 8 in 2024-03-12T03:49:04.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:03 smithi151 ceph-mon[32923]: pool 'ac83aab0-2794-4807-a391-8efea46d2288' is full (reached quota's max_objects: 10) 2024-03-12T03:49:04.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:03 smithi151 ceph-mon[32923]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-12T03:49:04.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:03 smithi151 ceph-mon[32923]: osdmap e818: 8 total, 8 up, 8 in 2024-03-12T03:49:05.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:04 smithi094 ceph-mon[29291]: pgmap v1791: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-12T03:49:05.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:04 smithi094 ceph-mon[40484]: pgmap v1791: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-12T03:49:05.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:04 smithi151 ceph-mon[32923]: pgmap v1791: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-12T03:49:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:06 smithi094 ceph-mon[29291]: pgmap v1793: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-12T03:49:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:06 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:49:06.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:06 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:06 smithi094 ceph-mon[40484]: pgmap v1793: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-12T03:49:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:06 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:49:06.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:06 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:06 smithi151 ceph-mon[32923]: pgmap v1793: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-12T03:49:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:06 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:49:06.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:06 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:08.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:08 smithi094 ceph-mon[29291]: pgmap v1794: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-12T03:49:08.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:08 smithi094 ceph-mon[40484]: pgmap v1794: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-12T03:49:08.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:08 smithi151 ceph-mon[32923]: pgmap v1794: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-12T03:49:09.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:49:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:49:10.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:10 smithi094 ceph-mon[29291]: pgmap v1795: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:49:10.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:10 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:10.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:49:10 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:49:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:49:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:10 smithi094 ceph-mon[40484]: pgmap v1795: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:49:10.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:10 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:10 smithi151 ceph-mon[32923]: pgmap v1795: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:49:10.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:10 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:12.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:12 smithi094 ceph-mon[29291]: pgmap v1796: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:49:12.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:12 smithi094 ceph-mon[40484]: pgmap v1796: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:49:12.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:12 smithi151 ceph-mon[32923]: pgmap v1796: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:49:14.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:14 smithi094 ceph-mon[29291]: pgmap v1797: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:49:14.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:14 smithi094 ceph-mon[40484]: pgmap v1797: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:49:14.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:14 smithi151 ceph-mon[32923]: pgmap v1797: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:49:16.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:16 smithi094 ceph-mon[29291]: pgmap v1798: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-12T03:49:16.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:16 smithi094 ceph-mon[40484]: pgmap v1798: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-12T03:49:16.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:16 smithi151 ceph-mon[32923]: pgmap v1798: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-12T03:49:18.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:18 smithi094 ceph-mon[29291]: pgmap v1799: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:18.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:18 smithi094 ceph-mon[40484]: pgmap v1799: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:18.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:18 smithi151 ceph-mon[32923]: pgmap v1799: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:19.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:49:19 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:49:20.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:20 smithi094 ceph-mon[29291]: pgmap v1800: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:20 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:20.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:20 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:20 smithi094 ceph-mon[40484]: pgmap v1800: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:20 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:20.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:20 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:20.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:49:20 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:49:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:49:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:20 smithi151 ceph-mon[32923]: pgmap v1800: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:20 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:20.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:20 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:22.964 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:22 smithi151 ceph-mon[32923]: pgmap v1801: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:49:23.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:22 smithi094 ceph-mon[29291]: pgmap v1801: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:49:23.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:22 smithi094 ceph-mon[40484]: pgmap v1801: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:49:25.090 INFO:tasks.workunit.client.0.smithi094.stderr:+ rados -p f98a828b-0804-4d73-9e78-2d8570d1ebe3 put threemore /etc/passwd 2024-03-12T03:49:25.130 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool set-quota f98a828b-0804-4d73-9e78-2d8570d1ebe3 max_bytes 0 2024-03-12T03:49:25.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:24 smithi094 ceph-mon[29291]: pgmap v1802: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:25.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:24 smithi094 ceph-mon[40484]: pgmap v1802: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:25.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:24 smithi151 ceph-mon[32923]: pgmap v1802: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:25.217 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.216+0000 7f8955668640 1 -- 172.21.15.94:0/2785117871 >> v1:172.21.15.94:6789/0 conn(0x7f89501002a0 legacy=0x7f8950100680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:25.217 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.216+0000 7f8955668640 1 -- 172.21.15.94:0/2785117871 shutdown_connections 2024-03-12T03:49:25.217 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.216+0000 7f8955668640 1 -- 172.21.15.94:0/2785117871 >> 172.21.15.94:0/2785117871 conn(0x7f8950076d00 msgr2=0x7f89500fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:49:25.218 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.216+0000 7f8955668640 1 -- 172.21.15.94:0/2785117871 shutdown_connections 2024-03-12T03:49:25.218 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.216+0000 7f8955668640 1 -- 172.21.15.94:0/2785117871 wait complete. 2024-03-12T03:49:25.218 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.216+0000 7f8955668640 1 Processor -- start 2024-03-12T03:49:25.221 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.216+0000 7f8955668640 1 -- start start 2024-03-12T03:49:25.221 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8955668640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f89501a5090 con 0x7f89501002a0 2024-03-12T03:49:25.221 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8955668640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f89501a62d0 con 0x7f8950108f20 2024-03-12T03:49:25.221 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8955668640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f89501a7510 con 0x7f895010ca90 2024-03-12T03:49:25.221 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f894effd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f89501002a0 0x7f89501055b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50178/0 (socket says 172.21.15.94:50178) 2024-03-12T03:49:25.221 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f894effd640 1 -- 172.21.15.94:0/1162198753 learned_addr learned my addr 172.21.15.94:0/1162198753 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:49:25.221 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912667178 0 0) 0x7f89501a7510 con 0x7f895010ca90 2024-03-12T03:49:25.221 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8938003520 con 0x7f895010ca90 2024-03-12T03:49:25.221 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174906229 0 0) 0x7f89501a5090 con 0x7f89501002a0 2024-03-12T03:49:25.221 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89501a7510 con 0x7f89501002a0 2024-03-12T03:49:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470253819 0 0) 0x7f89501a62d0 con 0x7f8950108f20 2024-03-12T03:49:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89501a5090 con 0x7f8950108f20 2024-03-12T03:49:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4226222276 0 0) 0x7f8938003520 con 0x7f895010ca90 2024-03-12T03:49:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89501a62d0 con 0x7f895010ca90 2024-03-12T03:49:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8948001b70 con 0x7f895010ca90 2024-03-12T03:49:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2795597175 0 0) 0x7f89501a7510 con 0x7f89501002a0 2024-03-12T03:49:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8938003520 con 0x7f89501002a0 2024-03-12T03:49:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8940004480 con 0x7f89501002a0 2024-03-12T03:49:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4223815306 0 0) 0x7f89501a5090 con 0x7f8950108f20 2024-03-12T03:49:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89501a7510 con 0x7f8950108f20 2024-03-12T03:49:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f89440013d0 con 0x7f8950108f20 2024-03-12T03:49:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3504820399 0 0) 0x7f89501a62d0 con 0x7f895010ca90 2024-03-12T03:49:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 >> v1:172.21.15.151:6789/0 conn(0x7f8950108f20 legacy=0x7f895019ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:25.223 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 >> v1:172.21.15.94:6789/0 conn(0x7f89501002a0 legacy=0x7f89501055b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:25.223 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89501a8750 con 0x7f895010ca90 2024-03-12T03:49:25.223 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.217+0000 7f8955668640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89501a7740 con 0x7f895010ca90 2024-03-12T03:49:25.223 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.218+0000 7f8955668640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89501a7c10 con 0x7f895010ca90 2024-03-12T03:49:25.223 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.218+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f8948003e00 con 0x7f895010ca90 2024-03-12T03:49:25.223 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.218+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f8948004c40 con 0x7f895010ca90 2024-03-12T03:49:25.223 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.218+0000 7f8955668640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893c005080 con 0x7f895010ca90 2024-03-12T03:49:25.223 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.219+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f8948003970 con 0x7f895010ca90 2024-03-12T03:49:25.223 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.219+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(818..818 src has 254..818) v4 ==== 8218+0+0 (unknown 2940364605 0 0) 0x7f8948088990 con 0x7f895010ca90 2024-03-12T03:49:25.223 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.220+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=819}) v3 -- 0x7f89501a62d0 con 0x7f895010ca90 2024-03-12T03:49:25.225 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.224+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f8948055a40 con 0x7f895010ca90 2024-03-12T03:49:25.402 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.400+0000 7f8955668640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f893c005370 con 0x7f895010ca90 2024-03-12T03:49:25.724 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.722+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.2 v1:172.21.15.94:6790/0 10 ==== osd_map(819..819 src has 254..819) v4 ==== 628+0+0 (unknown 2059159336 0 0) 0x7f8948004c40 con 0x7f895010ca90 2024-03-12T03:49:25.724 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.722+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=820}) v3 -- 0x7f8938003520 con 0x7f895010ca90 2024-03-12T03:49:25.728 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.727+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.2 v1:172.21.15.94:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v819) v1 ==== 217+0+0 (unknown 4190981243 0 0) 0x7f8948059490 con 0x7f895010ca90 2024-03-12T03:49:25.837 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:25.834+0000 7f8955668640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f893c000be0 con 0x7f895010ca90 2024-03-12T03:49:26.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:25 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1162198753' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:49:26.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:25 smithi094 ceph-mon[40484]: from='client.57452 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:49:26.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:25 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1162198753' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:49:26.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:25 smithi094 ceph-mon[29291]: from='client.57452 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:49:26.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:25 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1162198753' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:49:26.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:25 smithi151 ceph-mon[32923]: from='client.57452 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:49:26.732 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.729+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.2 v1:172.21.15.94:6790/0 12 ==== osd_map(820..820 src has 254..820) v4 ==== 628+0+0 (unknown 2242977377 0 0) 0x7f89480865b0 con 0x7f895010ca90 2024-03-12T03:49:26.732 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.729+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=821}) v3 -- 0x7f89501a5090 con 0x7f895010ca90 2024-03-12T03:49:26.735 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.733+0000 7f8937fff640 1 -- 172.21.15.94:0/1162198753 <== mon.2 v1:172.21.15.94:6790/0 13 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v820) v1 ==== 217+0+0 (unknown 3298066675 0 0) 0x7f894805dae0 con 0x7f895010ca90 2024-03-12T03:49:26.735 INFO:tasks.workunit.client.0.smithi094.stderr:set-quota max_bytes = 0 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 2024-03-12T03:49:26.739 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.737+0000 7f8955668640 1 -- 172.21.15.94:0/1162198753 >> v1:172.21.15.94:6800/2099808905 conn(0x7f893806ebe0 legacy=0x7f89380710a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:26.739 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.737+0000 7f8955668640 1 -- 172.21.15.94:0/1162198753 >> v1:172.21.15.94:6790/0 conn(0x7f895010ca90 legacy=0x7f89501a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:26.740 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.737+0000 7f8955668640 1 -- 172.21.15.94:0/1162198753 shutdown_connections 2024-03-12T03:49:26.740 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.737+0000 7f8955668640 1 -- 172.21.15.94:0/1162198753 >> 172.21.15.94:0/1162198753 conn(0x7f8950076d00 msgr2=0x7f8950107780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:49:26.740 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.737+0000 7f8955668640 1 -- 172.21.15.94:0/1162198753 shutdown_connections 2024-03-12T03:49:26.740 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.737+0000 7f8955668640 1 -- 172.21.15.94:0/1162198753 wait complete. 2024-03-12T03:49:26.749 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool set-quota f98a828b-0804-4d73-9e78-2d8570d1ebe3 max_objects 0 2024-03-12T03:49:26.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.829+0000 7f26e39f1640 1 -- 172.21.15.94:0/388608275 >> v1:172.21.15.94:6790/0 conn(0x7f26dc10a040 legacy=0x7f26dc10c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:26.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.829+0000 7f26e39f1640 1 -- 172.21.15.94:0/388608275 shutdown_connections 2024-03-12T03:49:26.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.829+0000 7f26e39f1640 1 -- 172.21.15.94:0/388608275 >> 172.21.15.94:0/388608275 conn(0x7f26dc075350 msgr2=0x7f26dc075760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:49:26.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.830+0000 7f26e39f1640 1 -- 172.21.15.94:0/388608275 shutdown_connections 2024-03-12T03:49:26.832 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.830+0000 7f26e39f1640 1 -- 172.21.15.94:0/388608275 wait complete. 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.830+0000 7f26e39f1640 1 Processor -- start 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.830+0000 7f26e39f1640 1 -- start start 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.830+0000 7f26e39f1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f26dc1a6190 con 0x7f26dc10a040 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.830+0000 7f26e39f1640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f26dc1a73d0 con 0x7f26dc10dbb0 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.830+0000 7f26e39f1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f26dc1a8610 con 0x7f26dc0ffbd0 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.830+0000 7f26e178d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f26dc0ffbd0 0x7f26dc106750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:51388/0 (socket says 172.21.15.94:51388) 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.830+0000 7f26e178d640 1 -- 172.21.15.94:0/3328156901 learned_addr learned my addr 172.21.15.94:0/3328156901 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447606815 0 0) 0x7f26dc1a6190 con 0x7f26dc10a040 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c8003520 con 0x7f26dc10a040 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087654182 0 0) 0x7f26dc1a8610 con 0x7f26dc0ffbd0 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26dc1a6190 con 0x7f26dc0ffbd0 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125155907 0 0) 0x7f26dc1a73d0 con 0x7f26dc10dbb0 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26dc1a8610 con 0x7f26dc10dbb0 2024-03-12T03:49:26.835 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 145211113 0 0) 0x7f26c8003520 con 0x7f26dc10a040 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26dc1a73d0 con 0x7f26dc10a040 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f26cc0044f0 con 0x7f26dc10a040 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3371230784 0 0) 0x7f26dc1a6190 con 0x7f26dc0ffbd0 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26c8003520 con 0x7f26dc0ffbd0 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f26d8001810 con 0x7f26dc0ffbd0 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2447006744 0 0) 0x7f26dc1a73d0 con 0x7f26dc10a040 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 >> v1:172.21.15.94:6790/0 conn(0x7f26dc0ffbd0 legacy=0x7f26dc106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 >> v1:172.21.15.151:6789/0 conn(0x7f26dc10dbb0 legacy=0x7f26dc1a4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc1a9850 con 0x7f26dc10a040 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26e39f1640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26dc1a8840 con 0x7f26dc10a040 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26e39f1640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26dc1a8d90 con 0x7f26dc10a040 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f26cc001160 con 0x7f26dc10a040 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.831+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f26cc004b60 con 0x7f26dc10a040 2024-03-12T03:49:26.836 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.832+0000 7f26e39f1640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c00050b0 con 0x7f26dc10a040 2024-03-12T03:49:26.837 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.832+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f26cc001b30 con 0x7f26dc10a040 2024-03-12T03:49:26.837 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.833+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(820..820 src has 254..820) v4 ==== 8218+0+0 (unknown 2823138451 0 0) 0x7f26cc0886e0 con 0x7f26dc10a040 2024-03-12T03:49:26.837 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.833+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=821}) v3 -- 0x7f26dc1a73d0 con 0x7f26dc10a040 2024-03-12T03:49:26.840 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:26.837+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f26cc055790 con 0x7f26dc10a040 2024-03-12T03:49:27.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:27.013+0000 7f26e39f1640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f26c00053a0 con 0x7f26dc10a040 2024-03-12T03:49:27.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:26 smithi094 ceph-mon[29291]: pgmap v1803: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:27.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:26 smithi094 ceph-mon[29291]: from='client.57452 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]': finished 2024-03-12T03:49:27.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:26 smithi094 ceph-mon[29291]: osdmap e819: 8 total, 8 up, 8 in 2024-03-12T03:49:27.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:26 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1162198753' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:49:27.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:26 smithi094 ceph-mon[29291]: from='client.57452 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:49:27.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:26 smithi094 ceph-mon[40484]: pgmap v1803: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:27.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:26 smithi094 ceph-mon[40484]: from='client.57452 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]': finished 2024-03-12T03:49:27.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:26 smithi094 ceph-mon[40484]: osdmap e819: 8 total, 8 up, 8 in 2024-03-12T03:49:27.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:26 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1162198753' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:49:27.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:26 smithi094 ceph-mon[40484]: from='client.57452 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:49:27.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:26 smithi151 ceph-mon[32923]: pgmap v1803: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:27.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:26 smithi151 ceph-mon[32923]: from='client.57452 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]': finished 2024-03-12T03:49:27.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:26 smithi151 ceph-mon[32923]: osdmap e819: 8 total, 8 up, 8 in 2024-03-12T03:49:27.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:26 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1162198753' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:49:27.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:26 smithi151 ceph-mon[32923]: from='client.57452 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-12T03:49:27.736 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:27.733+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v821) v1 ==== 221+0+0 (unknown 861213612 0 0) 0x7f26cc0591e0 con 0x7f26dc10a040 2024-03-12T03:49:27.736 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:27.733+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.0 v1:172.21.15.94:6789/0 11 ==== osd_map(821..821 src has 254..821) v4 ==== 628+0+0 (unknown 3021967305 0 0) 0x7f26cc086300 con 0x7f26dc10a040 2024-03-12T03:49:27.736 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:27.733+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=822}) v3 -- 0x7f26c8003520 con 0x7f26dc10a040 2024-03-12T03:49:27.845 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:27.843+0000 7f26e39f1640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f26c0005ae0 con 0x7f26dc10a040 2024-03-12T03:49:28.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:27 smithi094 ceph-mon[29291]: from='client.57452 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]': finished 2024-03-12T03:49:28.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:27 smithi094 ceph-mon[29291]: osdmap e820: 8 total, 8 up, 8 in 2024-03-12T03:49:28.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:27 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3328156901' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:49:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:27 smithi094 ceph-mon[40484]: from='client.57452 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]': finished 2024-03-12T03:49:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:27 smithi094 ceph-mon[40484]: osdmap e820: 8 total, 8 up, 8 in 2024-03-12T03:49:28.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:27 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3328156901' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:49:28.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:27 smithi151 ceph-mon[32923]: from='client.57452 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_bytes", "val": "0"}]': finished 2024-03-12T03:49:28.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:27 smithi151 ceph-mon[32923]: osdmap e820: 8 total, 8 up, 8 in 2024-03-12T03:49:28.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:27 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3328156901' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:49:28.742 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:28.739+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.0 v1:172.21.15.94:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 v822) v1 ==== 221+0+0 (unknown 3058177918 0 0) 0x7f26cc05d830 con 0x7f26dc10a040 2024-03-12T03:49:28.742 INFO:tasks.workunit.client.0.smithi094.stderr:set-quota max_objects = 0 for pool f98a828b-0804-4d73-9e78-2d8570d1ebe3 2024-03-12T03:49:28.742 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:28.739+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 <== mon.0 v1:172.21.15.94:6789/0 13 ==== osd_map(822..822 src has 254..822) v4 ==== 628+0+0 (unknown 444373008 0 0) 0x7f26cc0510d0 con 0x7f26dc10a040 2024-03-12T03:49:28.742 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:28.739+0000 7f26d27fc640 1 -- 172.21.15.94:0/3328156901 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=823}) v3 -- 0x7f26dc1a6190 con 0x7f26dc10a040 2024-03-12T03:49:28.746 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:28.743+0000 7f26e39f1640 1 -- 172.21.15.94:0/3328156901 >> v1:172.21.15.94:6800/2099808905 conn(0x7f26c806ebb0 legacy=0x7f26c8071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:28.746 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:28.743+0000 7f26e39f1640 1 -- 172.21.15.94:0/3328156901 >> v1:172.21.15.94:6789/0 conn(0x7f26dc10a040 legacy=0x7f26dc1a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:28.746 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:28.743+0000 7f26e39f1640 1 -- 172.21.15.94:0/3328156901 shutdown_connections 2024-03-12T03:49:28.746 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:28.743+0000 7f26e39f1640 1 -- 172.21.15.94:0/3328156901 >> 172.21.15.94:0/3328156901 conn(0x7f26dc075350 msgr2=0x7f26dc067b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:49:28.747 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:28.743+0000 7f26e39f1640 1 -- 172.21.15.94:0/3328156901 shutdown_connections 2024-03-12T03:49:28.747 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:28.744+0000 7f26e39f1640 1 -- 172.21.15.94:0/3328156901 wait complete. 2024-03-12T03:49:28.758 INFO:tasks.workunit.client.0.smithi094.stderr:+ sleep 30 2024-03-12T03:49:29.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:28 smithi094 ceph-mon[29291]: pgmap v1806: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:49:29.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3328156901' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]': finished 2024-03-12T03:49:29.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:28 smithi094 ceph-mon[29291]: osdmap e821: 8 total, 8 up, 8 in 2024-03-12T03:49:29.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:28 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3328156901' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:49:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:28 smithi094 ceph-mon[40484]: pgmap v1806: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:49:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3328156901' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]': finished 2024-03-12T03:49:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:28 smithi094 ceph-mon[40484]: osdmap e821: 8 total, 8 up, 8 in 2024-03-12T03:49:29.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:28 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3328156901' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:49:29.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:28 smithi151 ceph-mon[32923]: pgmap v1806: 188 pgs: 188 active+clean; 858 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:49:29.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3328156901' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]': finished 2024-03-12T03:49:29.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:28 smithi151 ceph-mon[32923]: osdmap e821: 8 total, 8 up, 8 in 2024-03-12T03:49:29.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:28 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3328156901' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]: dispatch 2024-03-12T03:49:29.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:49:29 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:49:30.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:29 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/3328156901' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]': finished 2024-03-12T03:49:30.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:29 smithi094 ceph-mon[29291]: osdmap e822: 8 total, 8 up, 8 in 2024-03-12T03:49:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:29 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/3328156901' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]': finished 2024-03-12T03:49:30.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:29 smithi094 ceph-mon[40484]: osdmap e822: 8 total, 8 up, 8 in 2024-03-12T03:49:30.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:29 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/3328156901' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "field": "max_objects", "val": "0"}]': finished 2024-03-12T03:49:30.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:29 smithi151 ceph-mon[32923]: osdmap e822: 8 total, 8 up, 8 in 2024-03-12T03:49:30.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:30 smithi094 ceph-mon[29291]: pgmap v1809: 188 pgs: 188 active+clean; 870 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-12T03:49:30.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:30 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:30.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:49:30 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:49:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:49:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:30 smithi094 ceph-mon[40484]: pgmap v1809: 188 pgs: 188 active+clean; 870 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-12T03:49:30.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:30 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:31.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:30 smithi151 ceph-mon[32923]: pgmap v1809: 188 pgs: 188 active+clean; 870 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-12T03:49:31.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:30 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:33.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:32 smithi094 ceph-mon[29291]: pgmap v1810: 188 pgs: 188 active+clean; 870 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 928 B/s rd, 2.2 KiB/s wr, 1 op/s 2024-03-12T03:49:33.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:32 smithi094 ceph-mon[40484]: pgmap v1810: 188 pgs: 188 active+clean; 870 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 928 B/s rd, 2.2 KiB/s wr, 1 op/s 2024-03-12T03:49:33.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:32 smithi151 ceph-mon[32923]: pgmap v1810: 188 pgs: 188 active+clean; 870 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 928 B/s rd, 2.2 KiB/s wr, 1 op/s 2024-03-12T03:49:35.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:34 smithi094 ceph-mon[29291]: pgmap v1811: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-12T03:49:35.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:34 smithi094 ceph-mon[40484]: pgmap v1811: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-12T03:49:35.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:34 smithi151 ceph-mon[32923]: pgmap v1811: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-12T03:49:36.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:35 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:36.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:35 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:36.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:35 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:37.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:36 smithi094 ceph-mon[29291]: pgmap v1812: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-12T03:49:37.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:36 smithi094 ceph-mon[40484]: pgmap v1812: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-12T03:49:37.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:36 smithi151 ceph-mon[32923]: pgmap v1812: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-12T03:49:39.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:38 smithi094 ceph-mon[29291]: pgmap v1813: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 538 B/s rd, 0 op/s 2024-03-12T03:49:39.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:38 smithi094 ceph-mon[40484]: pgmap v1813: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 538 B/s rd, 0 op/s 2024-03-12T03:49:39.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:38 smithi151 ceph-mon[32923]: pgmap v1813: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 538 B/s rd, 0 op/s 2024-03-12T03:49:39.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:49:39 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:49:40.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:40 smithi094 ceph-mon[29291]: pgmap v1814: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-03-12T03:49:40.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:40 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:40.960 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:49:40 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:49:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:49:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:40 smithi094 ceph-mon[40484]: pgmap v1814: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-03-12T03:49:40.960 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:40 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:41.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:40 smithi151 ceph-mon[32923]: pgmap v1814: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-03-12T03:49:41.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:40 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:43.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:42 smithi094 ceph-mon[29291]: pgmap v1815: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:43.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:42 smithi094 ceph-mon[40484]: pgmap v1815: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:43.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:42 smithi151 ceph-mon[32923]: pgmap v1815: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:45.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:44 smithi094 ceph-mon[29291]: pgmap v1816: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:45.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:44 smithi094 ceph-mon[40484]: pgmap v1816: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:45.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:44 smithi151 ceph-mon[32923]: pgmap v1816: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:47.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:46 smithi094 ceph-mon[29291]: pgmap v1817: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:49:47.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:46 smithi094 ceph-mon[40484]: pgmap v1817: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:49:47.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:46 smithi151 ceph-mon[32923]: pgmap v1817: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T03:49:48.115 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:47 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:49:48.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:47 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:49:48.116 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:47 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:49:48.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:47 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:49:48.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:47 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:49:48.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:47 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:49:48.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:47 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:49:48.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:47 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:49:48.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:47 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:49:49.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:48 smithi094 ceph-mon[29291]: pgmap v1818: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:48 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:49:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:48 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:49:49.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:48 smithi094 ceph-mon[29291]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:49:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:48 smithi094 ceph-mon[40484]: pgmap v1818: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:48 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:49:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:48 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:49:49.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:48 smithi094 ceph-mon[40484]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:49:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:48 smithi151 ceph-mon[32923]: pgmap v1818: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:48 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:49:49.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:48 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:49:49.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:48 smithi151 ceph-mon[32923]: from='mgr.34490 ' entity='mgr.y' 2024-03-12T03:49:49.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:49:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:49:50.783 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:49:50 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:49:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:49:51.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:50 smithi094 ceph-mon[29291]: pgmap v1819: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:51.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:50 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:51.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:50 smithi094 ceph-mon[29291]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:50 smithi094 ceph-mon[40484]: pgmap v1819: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:50 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:51.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:50 smithi094 ceph-mon[40484]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:50 smithi151 ceph-mon[32923]: pgmap v1819: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:50 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:51.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:50 smithi151 ceph-mon[32923]: from='mgr.34490 v1:172.21.15.94:0/783344958' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:53.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:52 smithi094 ceph-mon[29291]: pgmap v1820: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:53.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:52 smithi094 ceph-mon[40484]: pgmap v1820: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:53.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:52 smithi151 ceph-mon[32923]: pgmap v1820: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:55.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:54 smithi094 ceph-mon[29291]: pgmap v1821: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:55.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:54 smithi094 ceph-mon[40484]: pgmap v1821: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:55.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:54 smithi151 ceph-mon[32923]: pgmap v1821: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:57.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:56 smithi094 ceph-mon[40484]: pgmap v1822: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:57.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:56 smithi094 ceph-mon[29291]: pgmap v1822: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:57.217 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:56 smithi151 ceph-mon[32923]: pgmap v1822: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:58.759 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool delete f98a828b-0804-4d73-9e78-2d8570d1ebe3 f98a828b-0804-4d73-9e78-2d8570d1ebe3 --yes-i-really-really-mean-it 2024-03-12T03:49:58.856 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.850+0000 7f0a2aa14640 1 -- 172.21.15.94:0/301667398 >> v1:172.21.15.94:6790/0 conn(0x7f0a24102300 legacy=0x7f0a241026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:58.856 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.850+0000 7f0a2aa14640 1 -- 172.21.15.94:0/301667398 shutdown_connections 2024-03-12T03:49:58.856 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.850+0000 7f0a2aa14640 1 -- 172.21.15.94:0/301667398 >> 172.21.15.94:0/301667398 conn(0x7f0a24076d00 msgr2=0x7f0a240fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:49:58.856 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.850+0000 7f0a2aa14640 1 -- 172.21.15.94:0/301667398 shutdown_connections 2024-03-12T03:49:58.856 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.850+0000 7f0a2aa14640 1 -- 172.21.15.94:0/301667398 wait complete. 2024-03-12T03:49:58.856 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.850+0000 7f0a2aa14640 1 Processor -- start 2024-03-12T03:49:58.856 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.851+0000 7f0a2aa14640 1 -- start start 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.851+0000 7f0a2aa14640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a241a6270 con 0x7f0a2410a100 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.851+0000 7f0a2aa14640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a241a7470 con 0x7f0a2410dc10 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.851+0000 7f0a2aa14640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a241a86b0 con 0x7f0a24102300 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.851+0000 7f0a23fff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f0a24102300 0x7f0a241099b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:39812/0 (socket says 172.21.15.94:39812) 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.851+0000 7f0a23fff640 1 -- 172.21.15.94:0/1992640180 learned_addr learned my addr 172.21.15.94:0/1992640180 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.851+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859874900 0 0) 0x7f0a241a86b0 con 0x7f0a24102300 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.851+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0c003520 con 0x7f0a24102300 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.851+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699025683 0 0) 0x7f0a241a6270 con 0x7f0a2410a100 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.851+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a241a86b0 con 0x7f0a2410a100 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.851+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705888056 0 0) 0x7f0a241a7470 con 0x7f0a2410dc10 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.851+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a241a6270 con 0x7f0a2410dc10 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1034772600 0 0) 0x7f0a241a86b0 con 0x7f0a2410a100 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a241a7470 con 0x7f0a2410a100 2024-03-12T03:49:58.857 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2694735587 0 0) 0x7f0a0c003520 con 0x7f0a24102300 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a241a86b0 con 0x7f0a24102300 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f0a18001870 con 0x7f0a2410a100 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f0a140043d0 con 0x7f0a24102300 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 927994973 0 0) 0x7f0a241a6270 con 0x7f0a2410dc10 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a0c003520 con 0x7f0a2410dc10 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f0a1c0014b0 con 0x7f0a2410dc10 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4265308329 0 0) 0x7f0a241a7470 con 0x7f0a2410a100 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 >> v1:172.21.15.94:6790/0 conn(0x7f0a24102300 legacy=0x7f0a241099b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 >> v1:172.21.15.151:6789/0 conn(0x7f0a2410dc10 legacy=0x7f0a241a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a241a98f0 con 0x7f0a2410a100 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a2aa14640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a241a64a0 con 0x7f0a2410a100 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a2aa14640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a241a6970 con 0x7f0a2410a100 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f0a18001c60 con 0x7f0a2410a100 2024-03-12T03:49:58.858 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.852+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f0a18004a20 con 0x7f0a2410a100 2024-03-12T03:49:58.859 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.853+0000 7f0a2aa14640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a10005080 con 0x7f0a2410a100 2024-03-12T03:49:58.859 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.853+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f0a180038c0 con 0x7f0a2410a100 2024-03-12T03:49:58.859 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.854+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(822..822 src has 254..822) v4 ==== 8218+0+0 (unknown 1444726694 0 0) 0x7f0a180887c0 con 0x7f0a2410a100 2024-03-12T03:49:58.859 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.854+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=823}) v3 -- 0x7f0a241a7470 con 0x7f0a2410a100 2024-03-12T03:49:58.861 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:58.859+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f0a18055870 con 0x7f0a2410a100 2024-03-12T03:49:59.037 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:59.034+0000 7f0a2aa14640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pool2": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0a10005370 con 0x7f0a2410a100 2024-03-12T03:49:59.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:58 smithi094 ceph-mon[29291]: pgmap v1823: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:59.210 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:58 smithi094 ceph-mon[40484]: pgmap v1823: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:59.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:58 smithi151 ceph-mon[32923]: pgmap v1823: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:59.800 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:49:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug there is no tcmu-runner data available 2024-03-12T03:49:59.804 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:59.801+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pool2": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "yes_i_really_really_mean_it": true}]=0 pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' removed v823) v1 ==== 248+0+0 (unknown 274576642 0 0) 0x7f0a180592c0 con 0x7f0a2410a100 2024-03-12T03:49:59.804 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:59.802+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.0 v1:172.21.15.94:6789/0 11 ==== osd_map(823..823 src has 254..823) v4 ==== 296+0+0 (unknown 1533942099 0 0) 0x7f0a180863e0 con 0x7f0a2410a100 2024-03-12T03:49:59.804 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:59.802+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=824}) v3 -- 0x7f0a241a6270 con 0x7f0a2410a100 2024-03-12T03:49:59.919 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:59.917+0000 7f0a2aa14640 1 -- 172.21.15.94:0/1992640180 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pool2": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0a10005da0 con 0x7f0a2410a100 2024-03-12T03:49:59.920 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:59.917+0000 7f0a217fa640 1 -- 172.21.15.94:0/1992640180 <== mon.0 v1:172.21.15.94:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pool2": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "yes_i_really_really_mean_it": true}]=0 pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' does not exist v823) v1 ==== 255+0+0 (unknown 2235149202 0 0) 0x7f0a1805d910 con 0x7f0a2410a100 2024-03-12T03:49:59.920 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'f98a828b-0804-4d73-9e78-2d8570d1ebe3' does not exist 2024-03-12T03:49:59.922 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:59.919+0000 7f0a2aa14640 1 -- 172.21.15.94:0/1992640180 >> v1:172.21.15.94:6800/2099808905 conn(0x7f0a0c06ecb0 legacy=0x7f0a0c071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:59.922 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:59.919+0000 7f0a2aa14640 1 -- 172.21.15.94:0/1992640180 >> v1:172.21.15.94:6789/0 conn(0x7f0a2410a100 legacy=0x7f0a241a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:49:59.922 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:59.919+0000 7f0a2aa14640 1 -- 172.21.15.94:0/1992640180 shutdown_connections 2024-03-12T03:49:59.922 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:59.919+0000 7f0a2aa14640 1 -- 172.21.15.94:0/1992640180 >> 172.21.15.94:0/1992640180 conn(0x7f0a24076d00 msgr2=0x7f0a2410f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:49:59.922 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:59.919+0000 7f0a2aa14640 1 -- 172.21.15.94:0/1992640180 shutdown_connections 2024-03-12T03:49:59.922 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:49:59.919+0000 7f0a2aa14640 1 -- 172.21.15.94:0/1992640180 wait complete. 2024-03-12T03:49:59.930 INFO:tasks.workunit.client.0.smithi094.stderr:+ ceph osd pool delete ac83aab0-2794-4807-a391-8efea46d2288 ac83aab0-2794-4807-a391-8efea46d2288 --yes-i-really-really-mean-it 2024-03-12T03:50:00.015 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.010+0000 7f1bfcff1640 1 -- 172.21.15.94:0/2730438211 >> v1:172.21.15.94:6789/0 conn(0x7f1bf8108f20 legacy=0x7f1bf810b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:50:00.015 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.010+0000 7f1bfcff1640 1 -- 172.21.15.94:0/2730438211 shutdown_connections 2024-03-12T03:50:00.015 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.010+0000 7f1bfcff1640 1 -- 172.21.15.94:0/2730438211 >> 172.21.15.94:0/2730438211 conn(0x7f1bf8076d00 msgr2=0x7f1bf80fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:50:00.015 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.010+0000 7f1bfcff1640 1 -- 172.21.15.94:0/2730438211 shutdown_connections 2024-03-12T03:50:00.015 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.010+0000 7f1bfcff1640 1 -- 172.21.15.94:0/2730438211 wait complete. 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bfcff1640 1 Processor -- start 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bfcff1640 1 -- start start 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bfcff1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1bf81a4e10 con 0x7f1bf810ca90 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bfcff1640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1bf81a6050 con 0x7f1bf81002a0 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bfcff1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1bf81a7290 con 0x7f1bf8108f20 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bf6d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1bf810ca90 0x7f1bf81a3630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39240/0 (socket says 172.21.15.94:39240) 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1beffff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f1bf8108f20 0x7f1bf819ff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:39836/0 (socket says 172.21.15.94:39836) 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bf6d9d640 1 -- 172.21.15.94:0/4009454768 learned_addr learned my addr 172.21.15.94:0/4009454768 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340817042 0 0) 0x7f1bf81a7290 con 0x7f1bf8108f20 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be4003520 con 0x7f1bf8108f20 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951970943 0 0) 0x7f1bf81a4e10 con 0x7f1bf810ca90 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf81a7290 con 0x7f1bf810ca90 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020151042 0 0) 0x7f1bf81a6050 con 0x7f1bf81002a0 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf81a4e10 con 0x7f1bf81002a0 2024-03-12T03:50:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2418236194 0 0) 0x7f1be4003520 con 0x7f1bf8108f20 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bf81a6050 con 0x7f1bf8108f20 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1be0004480 con 0x7f1bf8108f20 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1715075580 0 0) 0x7f1bf81a7290 con 0x7f1bf810ca90 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1be4003520 con 0x7f1bf810ca90 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.011+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1bf0001d60 con 0x7f1bf810ca90 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.012+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 555741737 0 0) 0x7f1be4003520 con 0x7f1bf810ca90 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.012+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 >> v1:172.21.15.94:6790/0 conn(0x7f1bf8108f20 legacy=0x7f1bf819ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.012+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 >> v1:172.21.15.151:6789/0 conn(0x7f1bf81002a0 legacy=0x7f1bf8105330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.012+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf81a84d0 con 0x7f1bf810ca90 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.012+0000 7f1bfcff1640 1 -- 172.21.15.94:0/4009454768 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf81a5040 con 0x7f1bf810ca90 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.012+0000 7f1bfcff1640 1 -- 172.21.15.94:0/4009454768 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf81a5510 con 0x7f1bf810ca90 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.012+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 1938012057 0 0) 0x7f1bf0001fc0 con 0x7f1bf810ca90 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.012+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2736135423 0 0) 0x7f1bf0005f00 con 0x7f1bf810ca90 2024-03-12T03:50:00.017 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.013+0000 7f1bfcff1640 1 -- 172.21.15.94:0/4009454768 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd8005080 con 0x7f1bf810ca90 2024-03-12T03:50:00.018 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.013+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 22) v1 ==== 91776+0+0 (unknown 3475297846 0 0) 0x7f1bf0003c20 con 0x7f1bf810ca90 2024-03-12T03:50:00.018 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.014+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(823..823 src has 254..823) v4 ==== 7830+0+0 (unknown 1405546094 0 0) 0x7f1bf0089ae0 con 0x7f1bf810ca90 2024-03-12T03:50:00.018 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.014+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=824}) v3 -- 0x7f1be4003520 con 0x7f1bf810ca90 2024-03-12T03:50:00.021 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.018+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (unknown 1092875540 0 4211572577) 0x7f1bf0056d00 con 0x7f1bf810ca90 2024-03-12T03:50:00.196 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.193+0000 7f1bfcff1640 1 -- 172.21.15.94:0/4009454768 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pool2": "ac83aab0-2794-4807-a391-8efea46d2288", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1bd8005370 con 0x7f1bf810ca90 2024-03-12T03:50:00.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:49:59 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1992640180' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pool2": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:50:00.209 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:49:59 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1992640180' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pool2": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:50:00.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:49:59 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1992640180' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pool2": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:50:00.806 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.803+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pool2": "ac83aab0-2794-4807-a391-8efea46d2288", "yes_i_really_really_mean_it": true}]=0 pool 'ac83aab0-2794-4807-a391-8efea46d2288' removed v824) v1 ==== 248+0+0 (unknown 3009477295 0 0) 0x7f1bf005a750 con 0x7f1bf810ca90 2024-03-12T03:50:00.806 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.803+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.0 v1:172.21.15.94:6789/0 11 ==== osd_map(824..824 src has 254..824) v4 ==== 296+0+0 (unknown 3674853110 0 0) 0x7f1bf00522c0 con 0x7f1bf810ca90 2024-03-12T03:50:00.806 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.803+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=825}) v3 -- 0x7f1bf81a4e10 con 0x7f1bf810ca90 2024-03-12T03:50:00.818 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: ::ffff:172.21.15.151 - - [12/Mar/2024:03:50:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:50:00.920 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.918+0000 7f1bfcff1640 1 -- 172.21.15.94:0/4009454768 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pool2": "ac83aab0-2794-4807-a391-8efea46d2288", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1bd8005da0 con 0x7f1bf810ca90 2024-03-12T03:50:00.921 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.918+0000 7f1bef7fe640 1 -- 172.21.15.94:0/4009454768 <== mon.0 v1:172.21.15.94:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pool2": "ac83aab0-2794-4807-a391-8efea46d2288", "yes_i_really_really_mean_it": true}]=0 pool 'ac83aab0-2794-4807-a391-8efea46d2288' does not exist v824) v1 ==== 255+0+0 (unknown 2818153818 0 0) 0x7f1bf005eda0 con 0x7f1bf810ca90 2024-03-12T03:50:00.921 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'ac83aab0-2794-4807-a391-8efea46d2288' does not exist 2024-03-12T03:50:00.922 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.920+0000 7f1bfcff1640 1 -- 172.21.15.94:0/4009454768 >> v1:172.21.15.94:6800/2099808905 conn(0x7f1be406e910 legacy=0x7f1be4070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:50:00.923 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.920+0000 7f1bfcff1640 1 -- 172.21.15.94:0/4009454768 >> v1:172.21.15.94:6789/0 conn(0x7f1bf810ca90 legacy=0x7f1bf81a3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:50:00.923 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.920+0000 7f1bfcff1640 1 -- 172.21.15.94:0/4009454768 shutdown_connections 2024-03-12T03:50:00.923 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.920+0000 7f1bfcff1640 1 -- 172.21.15.94:0/4009454768 >> 172.21.15.94:0/4009454768 conn(0x7f1bf8076d00 msgr2=0x7f1bf810ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:50:00.923 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.920+0000 7f1bfcff1640 1 -- 172.21.15.94:0/4009454768 shutdown_connections 2024-03-12T03:50:00.923 INFO:tasks.workunit.client.0.smithi094.stderr:2024-03-12T03:50:00.920+0000 7f1bfcff1640 1 -- 172.21.15.94:0/4009454768 wait complete. 2024-03-12T03:50:00.931 INFO:tasks.workunit.client.0.smithi094.stderr:+ echo OK 2024-03-12T03:50:00.932 INFO:tasks.workunit.client.0.smithi094.stdout:OK 2024-03-12T03:50:00.933 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-12T03:50:00.933 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-12T03:50:01.001 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-03-12T03:50:01.002 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-12T03:50:01.069 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[29291]: pgmap v1824: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:01.069 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[29291]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:01.069 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1992640180' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pool2": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:50:01.069 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[29291]: osdmap e823: 8 total, 8 up, 8 in 2024-03-12T03:50:01.069 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/1992640180' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pool2": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:50:01.069 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[29291]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled; 1 pool(s) full 2024-03-12T03:50:01.070 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[29291]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-03-12T03:50:01.070 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[29291]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-12T03:50:01.070 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[29291]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-12T03:50:01.070 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[29291]: [WRN] POOL_FULL: 1 pool(s) full 2024-03-12T03:50:01.070 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[29291]: pool 'ac83aab0-2794-4807-a391-8efea46d2288' is full (running out of quota) 2024-03-12T03:50:01.070 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4009454768' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pool2": "ac83aab0-2794-4807-a391-8efea46d2288", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:50:01.070 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[40484]: pgmap v1824: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:01.070 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[40484]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:01.070 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1992640180' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pool2": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:50:01.070 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[40484]: osdmap e823: 8 total, 8 up, 8 in 2024-03-12T03:50:01.070 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/1992640180' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pool2": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:50:01.071 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[40484]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled; 1 pool(s) full 2024-03-12T03:50:01.071 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[40484]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-03-12T03:50:01.071 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[40484]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-12T03:50:01.071 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[40484]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-12T03:50:01.071 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[40484]: [WRN] POOL_FULL: 1 pool(s) full 2024-03-12T03:50:01.071 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[40484]: pool 'ac83aab0-2794-4807-a391-8efea46d2288' is full (running out of quota) 2024-03-12T03:50:01.071 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:00 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4009454768' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pool2": "ac83aab0-2794-4807-a391-8efea46d2288", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:50:01.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:00 smithi151 ceph-mon[32923]: pgmap v1824: 188 pgs: 188 active+clean; 870 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:01.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:00 smithi151 ceph-mon[32923]: from='client.24460 v1:172.21.15.151:0/2396130126' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:01.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1992640180' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pool2": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:50:01.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:00 smithi151 ceph-mon[32923]: osdmap e823: 8 total, 8 up, 8 in 2024-03-12T03:50:01.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/1992640180' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "pool2": "f98a828b-0804-4d73-9e78-2d8570d1ebe3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:50:01.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:00 smithi151 ceph-mon[32923]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled; 1 pool(s) full 2024-03-12T03:50:01.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:00 smithi151 ceph-mon[32923]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-03-12T03:50:01.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:00 smithi151 ceph-mon[32923]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-12T03:50:01.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:00 smithi151 ceph-mon[32923]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-12T03:50:01.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:00 smithi151 ceph-mon[32923]: [WRN] POOL_FULL: 1 pool(s) full 2024-03-12T03:50:01.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:00 smithi151 ceph-mon[32923]: pool 'ac83aab0-2794-4807-a391-8efea46d2288' is full (running out of quota) 2024-03-12T03:50:01.216 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:00 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4009454768' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pool2": "ac83aab0-2794-4807-a391-8efea46d2288", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:50:01.327 DEBUG:teuthology.parallel:result is None 2024-03-12T03:50:01.328 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-12T03:50:01.394 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-12T03:50:01.394 DEBUG:teuthology.orchestra.run.smithi094:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-12T03:50:01.453 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-12T03:50:01.453 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-12T03:50:01.463 INFO:tasks.cephadm:Teardown begin 2024-03-12T03:50:01.463 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:50:01.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:50:01.563 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-12T03:50:01.564 DEBUG:teuthology.orchestra.run.smithi094:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T03:50:01.583 DEBUG:teuthology.orchestra.run.smithi151:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T03:50:01.619 INFO:tasks.cephadm:Stopping all daemons... 2024-03-12T03:50:01.620 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-12T03:50:01.620 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.a 2024-03-12T03:50:01.944 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:01 smithi094 ceph-mon[40484]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-12T03:50:01.945 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4009454768' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pool2": "ac83aab0-2794-4807-a391-8efea46d2288", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:50:01.945 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:01 smithi094 ceph-mon[40484]: osdmap e824: 8 total, 8 up, 8 in 2024-03-12T03:50:01.945 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:01 smithi094 ceph-mon[40484]: from='client.? v1:172.21.15.94:0/4009454768' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pool2": "ac83aab0-2794-4807-a391-8efea46d2288", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:50:01.945 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:01 smithi094 systemd[1]: Stopping Ceph mon.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:50:01.945 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:01 smithi094 ceph-mon[29291]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-12T03:50:01.945 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4009454768' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pool2": "ac83aab0-2794-4807-a391-8efea46d2288", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:50:01.945 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:01 smithi094 ceph-mon[29291]: osdmap e824: 8 total, 8 up, 8 in 2024-03-12T03:50:01.945 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:01 smithi094 ceph-mon[29291]: from='client.? v1:172.21.15.94:0/4009454768' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pool2": "ac83aab0-2794-4807-a391-8efea46d2288", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:50:02.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:01 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a[29287]: 2024-03-12T03:50:01.941+0000 7f068fcb8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:50:02.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:01 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a[29287]: 2024-03-12T03:50:01.941+0000 7f068fcb8700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-12T03:50:02.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:01 smithi094 podman[155895]: 2024-03-12 03:50:01.973571754 +0000 UTC m=+0.175267929 container died 57104d25fd78692e58cef1a988ee5927b17b28071a2e5040bcb367321446dfb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-12T03:50:02.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:01 smithi151 ceph-mon[32923]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-12T03:50:02.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4009454768' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pool2": "ac83aab0-2794-4807-a391-8efea46d2288", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:50:02.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:01 smithi151 ceph-mon[32923]: osdmap e824: 8 total, 8 up, 8 in 2024-03-12T03:50:02.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:01 smithi151 ceph-mon[32923]: from='client.? v1:172.21.15.94:0/4009454768' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "ac83aab0-2794-4807-a391-8efea46d2288", "pool2": "ac83aab0-2794-4807-a391-8efea46d2288", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:50:02.613 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:02 smithi094 podman[155895]: 2024-03-12 03:50:02.418253407 +0000 UTC m=+0.619949619 container cleanup 57104d25fd78692e58cef1a988ee5927b17b28071a2e5040bcb367321446dfb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-12T03:50:02.613 INFO:journalctl@ceph.mon.a.smithi094.stdout:Mar 12 03:50:02 smithi094 bash[155895]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-a 2024-03-12T03:50:02.827 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.a.service' 2024-03-12T03:50:02.867 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:50:02.867 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-12T03:50:02.868 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-12T03:50:02.868 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.c 2024-03-12T03:50:03.199 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:02 smithi094 systemd[1]: Stopping Ceph mon.c for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:50:03.459 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:03 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-c[40480]: 2024-03-12T03:50:03.197+0000 7ffb35a2a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:50:03.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:03 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-c[40480]: 2024-03-12T03:50:03.197+0000 7ffb35a2a700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-12T03:50:03.460 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:03 smithi094 podman[156017]: 2024-03-12 03:50:03.341068756 +0000 UTC m=+0.252506503 container died 09f920794c6cb263d315fb2f233703155436059e19b2c6ec85a5cd643a7c1392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-c, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:50:03.460 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:50:03 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:50:03] ENGINE Bus STOPPING 2024-03-12T03:50:03.460 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:50:03 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:50:03] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:50:03.460 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:50:03 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:50:03] ENGINE Bus STOPPED 2024-03-12T03:50:03.460 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:50:03 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:50:03] ENGINE Bus STARTING 2024-03-12T03:50:03.460 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:50:03 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:50:03] ENGINE Serving on http://:::9283 2024-03-12T03:50:03.460 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:50:03 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: [12/Mar/2024:03:50:03] ENGINE Bus STARTED 2024-03-12T03:50:03.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:03 smithi094 podman[156017]: 2024-03-12 03:50:03.70742715 +0000 UTC m=+0.618864897 container cleanup 09f920794c6cb263d315fb2f233703155436059e19b2c6ec85a5cd643a7c1392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-c, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:50:03.959 INFO:journalctl@ceph.mon.c.smithi094.stdout:Mar 12 03:50:03 smithi094 bash[156017]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-c 2024-03-12T03:50:04.179 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.c.service' 2024-03-12T03:50:04.216 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:50:04.216 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-12T03:50:04.216 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-12T03:50:04.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.b 2024-03-12T03:50:04.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:04 smithi151 systemd[1]: Stopping Ceph mon.b for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:50:05.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:04 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-b[32916]: 2024-03-12T03:50:04.822+0000 7fcd245de700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:50:05.214 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:04 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-b[32916]: 2024-03-12T03:50:04.822+0000 7fcd245de700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-12T03:50:05.215 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:05 smithi151 podman[75107]: 2024-03-12 03:50:05.007278089 +0000 UTC m=+0.594296910 container died 5f664b44d1d7bff77c00d729ff0229587d238140b4f14627a95255c59b9657c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-12T03:50:05.714 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:05 smithi151 podman[75107]: 2024-03-12 03:50:05.481891841 +0000 UTC m=+1.068910661 container cleanup 5f664b44d1d7bff77c00d729ff0229587d238140b4f14627a95255c59b9657c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:50:05.715 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 12 03:50:05 smithi151 bash[75107]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-mon-b 2024-03-12T03:50:05.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@mon.b.service' 2024-03-12T03:50:05.908 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:50:05.908 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-12T03:50:05.908 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-12T03:50:05.908 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@mgr.y 2024-03-12T03:50:06.206 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:50:05 smithi094 systemd[1]: Stopping Ceph mgr.y for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:50:06.459 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:50:06 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:50:06.203+0000 7fd9de884700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:50:06.460 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:50:06 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y[29524]: 2024-03-12T03:50:06.203+0000 7fd9de884700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-12T03:50:06.460 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:50:06 smithi094 podman[156149]: 2024-03-12 03:50:06.236169296 +0000 UTC m=+0.140015340 container died c4e8b75fe5c967531eb5b4c68f00005ae3da6ead27ab6e2aec4e89b68788ba00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-12T03:50:06.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:50:06 smithi094 podman[156149]: 2024-03-12 03:50:06.739279419 +0000 UTC m=+0.643125453 container cleanup c4e8b75fe5c967531eb5b4c68f00005ae3da6ead27ab6e2aec4e89b68788ba00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-12T03:50:06.959 INFO:journalctl@ceph.mgr.y.smithi094.stdout:Mar 12 03:50:06 smithi094 bash[156149]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-y 2024-03-12T03:50:07.160 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-12T03:50:07.195 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:50:07.196 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-12T03:50:07.196 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-12T03:50:07.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@mgr.x 2024-03-12T03:50:07.536 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:50:07 smithi151 systemd[1]: Stopping Ceph mgr.x for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:50:07.813 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:50:07 smithi151 podman[75230]: 2024-03-12 03:50:07.534373089 +0000 UTC m=+0.143124835 container died 804d02ce0a6662750972429604f8fce21d6c5b72ddc3dbe0117990afa8000884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-12T03:50:08.147 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:50:07 smithi151 podman[75230]: 2024-03-12 03:50:07.894725124 +0000 UTC m=+0.503476913 container cleanup 804d02ce0a6662750972429604f8fce21d6c5b72ddc3dbe0117990afa8000884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:50:08.147 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 12 03:50:07 smithi151 bash[75230]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-mgr-x 2024-03-12T03:50:08.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-12T03:50:08.303 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:50:08.303 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-12T03:50:08.303 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-12T03:50:08.303 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.0 2024-03-12T03:50:08.605 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:08 smithi094 systemd[1]: Stopping Ceph osd.0 for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:50:08.959 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:08 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0[44487]: 2024-03-12T03:50:08.601+0000 7f3c13738700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:50:08.959 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:08 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0[44487]: 2024-03-12T03:50:08.601+0000 7f3c13738700 -1 osd.0 824 *** Got signal Terminated *** 2024-03-12T03:50:08.959 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:08 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0[44487]: 2024-03-12T03:50:08.601+0000 7f3c13738700 -1 osd.0 824 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:50:14.280 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:14 smithi094 podman[156271]: 2024-03-12 03:50:14.01179153 +0000 UTC m=+5.518848108 container died 38695d717240b6ded13e32337a11ea7255ff6e2745ffd6a1b61caeeeb90df5cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-12T03:50:14.640 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:14 smithi094 podman[156271]: 2024-03-12 03:50:14.378190827 +0000 UTC m=+5.885247405 container cleanup 38695d717240b6ded13e32337a11ea7255ff6e2745ffd6a1b61caeeeb90df5cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:50:14.640 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:14 smithi094 bash[156271]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0 2024-03-12T03:50:14.959 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:14 smithi094 podman[156288]: 2024-03-12 03:50:14.638290268 +0000 UTC m=+0.616655628 container remove 38695d717240b6ded13e32337a11ea7255ff6e2745ffd6a1b61caeeeb90df5cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-2b0101e, ceph=True) 2024-03-12T03:50:15.311 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:14 smithi094 podman[156380]: 2024-03-12 03:50:14.901405603 +0000 UTC m=+0.017974676 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:50:15.311 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:15 smithi094 podman[156380]: 2024-03-12 03:50:15.006241633 +0000 UTC m=+0.122810752 container create 92b36896a96fef855862ca4a9024c84053ca10bc9c0ca1a86f9639d2bb7dd039 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0-deactivate, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:50:15.311 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:15 smithi094 podman[156380]: 2024-03-12 03:50:15.234124052 +0000 UTC m=+0.350693125 container init 92b36896a96fef855862ca4a9024c84053ca10bc9c0ca1a86f9639d2bb7dd039 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-12T03:50:15.312 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:15 smithi094 podman[156380]: 2024-03-12 03:50:15.237191161 +0000 UTC m=+0.353760244 container start 92b36896a96fef855862ca4a9024c84053ca10bc9c0ca1a86f9639d2bb7dd039 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e) 2024-03-12T03:50:15.312 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:15 smithi094 podman[156380]: 2024-03-12 03:50:15.309216359 +0000 UTC m=+0.425785443 container attach 92b36896a96fef855862ca4a9024c84053ca10bc9c0ca1a86f9639d2bb7dd039 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS) 2024-03-12T03:50:15.709 INFO:journalctl@ceph.osd.0.smithi094.stdout:Mar 12 03:50:15 smithi094 podman[156380]: 2024-03-12 03:50:15.52585391 +0000 UTC m=+0.642422975 container died 92b36896a96fef855862ca4a9024c84053ca10bc9c0ca1a86f9639d2bb7dd039 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:50:16.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.0.service' 2024-03-12T03:50:16.137 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:50:16.137 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-12T03:50:16.137 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-12T03:50:16.137 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.1 2024-03-12T03:50:16.367 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:16 smithi094 systemd[1]: Stopping Ceph osd.1 for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:50:16.709 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1[50221]: 2024-03-12T03:50:16.500+0000 7f2b450f9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:50:16.709 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1[50221]: 2024-03-12T03:50:16.500+0000 7f2b450f9700 -1 osd.1 824 *** Got signal Terminated *** 2024-03-12T03:50:16.710 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:16 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1[50221]: 2024-03-12T03:50:16.500+0000 7f2b450f9700 -1 osd.1 824 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:50:22.188 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:21 smithi094 podman[156486]: 2024-03-12 03:50:21.925430241 +0000 UTC m=+5.556608432 container died f717a8b83319da8028c3bfc02e9cff363037d24aeb3ad1a19fac259a46c086a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-12T03:50:22.459 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:22 smithi094 podman[156486]: 2024-03-12 03:50:22.284414881 +0000 UTC m=+5.915593095 container cleanup f717a8b83319da8028c3bfc02e9cff363037d24aeb3ad1a19fac259a46c086a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e) 2024-03-12T03:50:22.459 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:22 smithi094 bash[156486]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1 2024-03-12T03:50:22.948 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:22 smithi094 podman[156515]: 2024-03-12 03:50:22.603155968 +0000 UTC m=+0.676763261 container remove f717a8b83319da8028c3bfc02e9cff363037d24aeb3ad1a19fac259a46c086a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-12T03:50:23.209 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:22 smithi094 podman[156608]: 2024-03-12 03:50:22.849346045 +0000 UTC m=+0.017619884 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:50:23.209 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:22 smithi094 podman[156608]: 2024-03-12 03:50:22.954309908 +0000 UTC m=+0.122583718 container create ddc9eaea78cc15ec3b0bfea9d015b1936350aa5d4c6ca5bd28ca50879a006460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-12T03:50:23.459 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:23 smithi094 podman[156608]: 2024-03-12 03:50:23.207277062 +0000 UTC m=+0.375550880 container init ddc9eaea78cc15ec3b0bfea9d015b1936350aa5d4c6ca5bd28ca50879a006460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:50:23.460 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:23 smithi094 podman[156608]: 2024-03-12 03:50:23.210414975 +0000 UTC m=+0.378688786 container start ddc9eaea78cc15ec3b0bfea9d015b1936350aa5d4c6ca5bd28ca50879a006460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-12T03:50:23.460 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:23 smithi094 podman[156608]: 2024-03-12 03:50:23.324153749 +0000 UTC m=+0.492427562 container attach ddc9eaea78cc15ec3b0bfea9d015b1936350aa5d4c6ca5bd28ca50879a006460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-2b0101e) 2024-03-12T03:50:23.854 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:23 smithi094 podman[156608]: 2024-03-12 03:50:23.49504513 +0000 UTC m=+0.663318954 container died ddc9eaea78cc15ec3b0bfea9d015b1936350aa5d4c6ca5bd28ca50879a006460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-12T03:50:24.107 INFO:journalctl@ceph.osd.1.smithi094.stdout:Mar 12 03:50:24 smithi094 podman[156634]: 2024-03-12 03:50:24.086931532 +0000 UTC m=+0.582866091 container remove ddc9eaea78cc15ec3b0bfea9d015b1936350aa5d4c6ca5bd28ca50879a006460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-1-deactivate, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-12T03:50:24.113 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.1.service' 2024-03-12T03:50:24.156 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:50:24.156 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-12T03:50:24.156 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-12T03:50:24.157 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.2 2024-03-12T03:50:24.459 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:24 smithi094 systemd[1]: Stopping Ceph osd.2 for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:50:24.959 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:24 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2[56245]: 2024-03-12T03:50:24.539+0000 7fcfc67db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:50:24.960 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:24 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2[56245]: 2024-03-12T03:50:24.539+0000 7fcfc67db700 -1 osd.2 824 *** Got signal Terminated *** 2024-03-12T03:50:24.960 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:24 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2[56245]: 2024-03-12T03:50:24.539+0000 7fcfc67db700 -1 osd.2 824 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:50:30.184 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:29 smithi094 podman[156715]: 2024-03-12 03:50:29.893913805 +0000 UTC m=+5.519451772 container died cabe8627e27c24de53a7a961fa6bb314336b136fda6fb6cc11a4752567d80550 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-12T03:50:30.459 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:30 smithi094 podman[156715]: 2024-03-12 03:50:30.282477185 +0000 UTC m=+5.908015151 container cleanup cabe8627e27c24de53a7a961fa6bb314336b136fda6fb6cc11a4752567d80550 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-12T03:50:30.459 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:30 smithi094 bash[156715]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2 2024-03-12T03:50:30.895 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:30 smithi094 podman[156732]: 2024-03-12 03:50:30.542430783 +0000 UTC m=+0.645331214 container remove cabe8627e27c24de53a7a961fa6bb314336b136fda6fb6cc11a4752567d80550 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=) 2024-03-12T03:50:31.207 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:30 smithi094 podman[156818]: 2024-03-12 03:50:30.796265147 +0000 UTC m=+0.017531969 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:50:31.207 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:30 smithi094 podman[156818]: 2024-03-12 03:50:30.910141311 +0000 UTC m=+0.131408144 container create 84ee00a7ce0e646eb8c0a513d6dbe055a43fde6e5b193c0411000688e1fdb4ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-12T03:50:31.207 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:31 smithi094 podman[156818]: 2024-03-12 03:50:31.121674969 +0000 UTC m=+0.342941804 container init 84ee00a7ce0e646eb8c0a513d6dbe055a43fde6e5b193c0411000688e1fdb4ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-12T03:50:31.207 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:31 smithi094 podman[156818]: 2024-03-12 03:50:31.124835159 +0000 UTC m=+0.346101985 container start 84ee00a7ce0e646eb8c0a513d6dbe055a43fde6e5b193c0411000688e1fdb4ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:50:31.207 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:31 smithi094 podman[156818]: 2024-03-12 03:50:31.205107274 +0000 UTC m=+0.426374114 container attach 84ee00a7ce0e646eb8c0a513d6dbe055a43fde6e5b193c0411000688e1fdb4ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-03-12T03:50:31.459 INFO:journalctl@ceph.osd.2.smithi094.stdout:Mar 12 03:50:31 smithi094 podman[156818]: 2024-03-12 03:50:31.410241645 +0000 UTC m=+0.631508467 container died 84ee00a7ce0e646eb8c0a513d6dbe055a43fde6e5b193c0411000688e1fdb4ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-12T03:50:32.041 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.2.service' 2024-03-12T03:50:32.084 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:50:32.085 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-12T03:50:32.085 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-12T03:50:32.085 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.3 2024-03-12T03:50:32.414 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:32 smithi094 systemd[1]: Stopping Ceph osd.3 for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:50:32.709 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:32 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3[62481]: 2024-03-12T03:50:32.411+0000 7f6859cbd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:50:32.709 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:32 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3[62481]: 2024-03-12T03:50:32.411+0000 7f6859cbd700 -1 osd.3 824 *** Got signal Terminated *** 2024-03-12T03:50:32.710 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:32 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3[62481]: 2024-03-12T03:50:32.411+0000 7f6859cbd700 -1 osd.3 824 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:50:35.884 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:35 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:35.471+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:36.148 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:36 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:36.009+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:36.714 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:36 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:36.439+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:36.959 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:36 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3[62481]: 2024-03-12T03:50:36.520+0000 7f68562b1700 -1 osd.3 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.095093+0000 front 2024-03-12T03:50:12.095112+0000 (oldest deadline 2024-03-12T03:50:35.595280+0000) 2024-03-12T03:50:37.386 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:37 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:37.028+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:37.714 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:37 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:37.420+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:37.715 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:37 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:37.384+0000 7f74a42fb700 -1 osd.4 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:11.191076+0000 front 2024-03-12T03:50:11.191278+0000 (oldest deadline 2024-03-12T03:50:36.491131+0000) 2024-03-12T03:50:38.057 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:37 smithi094 podman[156925]: 2024-03-12 03:50:37.735877908 +0000 UTC m=+5.431660793 container died cd694cf384ef88d83c7c3afece1082000ea8bb99527ed577a5fef5b3595d1a55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-12T03:50:38.382 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:38 smithi094 podman[156925]: 2024-03-12 03:50:38.146079788 +0000 UTC m=+5.841862672 container cleanup cd694cf384ef88d83c7c3afece1082000ea8bb99527ed577a5fef5b3595d1a55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:50:38.383 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:38 smithi094 bash[156925]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3 2024-03-12T03:50:38.395 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:38.058+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:38.709 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:38 smithi094 podman[156942]: 2024-03-12 03:50:38.381039999 +0000 UTC m=+0.644025915 container remove cd694cf384ef88d83c7c3afece1082000ea8bb99527ed577a5fef5b3595d1a55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:50:38.714 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:38.411+0000 7f74a42fb700 -1 osd.4 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:11.191076+0000 front 2024-03-12T03:50:11.191278+0000 (oldest deadline 2024-03-12T03:50:36.491131+0000) 2024-03-12T03:50:38.714 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:38 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:38.392+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:39.022 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:38 smithi094 podman[157029]: 2024-03-12 03:50:38.672646623 +0000 UTC m=+0.017532752 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:50:39.022 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:38 smithi094 podman[157029]: 2024-03-12 03:50:38.807113894 +0000 UTC m=+0.152000050 container create 34289d089ec0517cd93cbc8e89fde186135133b66236deabd741a20996a2624b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-12T03:50:39.023 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:39 smithi094 podman[157029]: 2024-03-12 03:50:39.017960724 +0000 UTC m=+0.362846844 container init 34289d089ec0517cd93cbc8e89fde186135133b66236deabd741a20996a2624b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-12T03:50:39.023 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:39 smithi094 podman[157029]: 2024-03-12 03:50:39.020944756 +0000 UTC m=+0.365830867 container start 34289d089ec0517cd93cbc8e89fde186135133b66236deabd741a20996a2624b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-2b0101e, ceph=True) 2024-03-12T03:50:39.312 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:39 smithi094 podman[157029]: 2024-03-12 03:50:39.118247448 +0000 UTC m=+0.463133590 container attach 34289d089ec0517cd93cbc8e89fde186135133b66236deabd741a20996a2624b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-12T03:50:39.370 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:39 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:39.094+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:39.370 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:39 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:39.158+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:39.656 INFO:journalctl@ceph.osd.3.smithi094.stdout:Mar 12 03:50:39 smithi094 podman[157029]: 2024-03-12 03:50:39.310628412 +0000 UTC m=+0.655514524 container died 34289d089ec0517cd93cbc8e89fde186135133b66236deabd741a20996a2624b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-03-12T03:50:39.714 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:39 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:39.367+0000 7f74a42fb700 -1 osd.4 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:11.191076+0000 front 2024-03-12T03:50:11.191278+0000 (oldest deadline 2024-03-12T03:50:36.491131+0000) 2024-03-12T03:50:39.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:39 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:39.403+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:39.972 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.3.service' 2024-03-12T03:50:40.016 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:50:40.016 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-12T03:50:40.016 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-12T03:50:40.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.4 2024-03-12T03:50:40.357 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:40 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:40.113+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:40.357 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:40 smithi151 systemd[1]: Stopping Ceph osd.4 for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:50:40.357 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:40 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:40.130+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:40.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:40 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:40.421+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:40.715 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:40 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:40.354+0000 7f74a42fb700 -1 osd.4 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:11.191076+0000 front 2024-03-12T03:50:11.191278+0000 (oldest deadline 2024-03-12T03:50:36.491131+0000) 2024-03-12T03:50:40.715 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:40 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:40.395+0000 7f74a7d07700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:50:40.715 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:40 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:40.395+0000 7f74a7d07700 -1 osd.4 824 *** Got signal Terminated *** 2024-03-12T03:50:40.715 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:40 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:40.395+0000 7f74a7d07700 -1 osd.4 824 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:50:41.381 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:41.094+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:41.382 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:41.113+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:41.714 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:41.447+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:41.715 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:41 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:41.378+0000 7f74a42fb700 -1 osd.4 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:11.191076+0000 front 2024-03-12T03:50:11.191278+0000 (oldest deadline 2024-03-12T03:50:36.491131+0000) 2024-03-12T03:50:42.401 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:42 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:42.073+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:42.402 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:42 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:42.083+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:42.714 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:42 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:42.399+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:42.715 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:42 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:42.410+0000 7f74a42fb700 -1 osd.4 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:11.191076+0000 front 2024-03-12T03:50:11.191278+0000 (oldest deadline 2024-03-12T03:50:36.491131+0000) 2024-03-12T03:50:42.715 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:42 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:42.410+0000 7f74a42fb700 -1 osd.4 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:16.491832+0000 front 2024-03-12T03:50:16.491716+0000 (oldest deadline 2024-03-12T03:50:42.391735+0000) 2024-03-12T03:50:43.383 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:43.090+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:43.384 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:43.090+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:43.384 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:43.093+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:43.384 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:43.093+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:18.142176+0000 front 2024-03-12T03:50:18.142132+0000 (oldest deadline 2024-03-12T03:50:42.142480+0000) 2024-03-12T03:50:43.384 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:50:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:50:43.215Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.94:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.94:8765: connect: connection refused" 2024-03-12T03:50:43.385 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:50:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:50:43.218Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.94:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.94:8765: connect: connection refused" 2024-03-12T03:50:43.385 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:50:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:50:43.218Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.94:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.94:8765: connect: connection refused" 2024-03-12T03:50:43.385 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:50:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:50:43.219Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.94:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.94:8765: connect: connection refused" 2024-03-12T03:50:43.385 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:50:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:50:43.219Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.94:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.94:8765: connect: connection refused" 2024-03-12T03:50:43.714 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:43.381+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:43.715 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:43.381+0000 7f74a42fb700 -1 osd.4 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:11.191076+0000 front 2024-03-12T03:50:11.191278+0000 (oldest deadline 2024-03-12T03:50:36.491131+0000) 2024-03-12T03:50:43.715 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:43 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:43.381+0000 7f74a42fb700 -1 osd.4 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:16.491832+0000 front 2024-03-12T03:50:16.491716+0000 (oldest deadline 2024-03-12T03:50:42.391735+0000) 2024-03-12T03:50:44.337 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:44 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:44.083+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:44.337 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:44 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:44.083+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:18.142176+0000 front 2024-03-12T03:50:18.142132+0000 (oldest deadline 2024-03-12T03:50:42.142480+0000) 2024-03-12T03:50:44.337 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:44 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:44.126+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:44.337 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:44 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:44.126+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:44.714 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:44 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:44.334+0000 7f74a42fb700 -1 osd.4 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:11.191076+0000 front 2024-03-12T03:50:11.191278+0000 (oldest deadline 2024-03-12T03:50:36.491131+0000) 2024-03-12T03:50:44.715 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:44 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:44.334+0000 7f74a42fb700 -1 osd.4 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:16.491832+0000 front 2024-03-12T03:50:16.491716+0000 (oldest deadline 2024-03-12T03:50:42.391735+0000) 2024-03-12T03:50:44.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:44 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:44.367+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:44.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:44 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:44.367+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:45.358 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:45 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:45.322+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:45.359 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:45 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:45.322+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:45.359 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:45 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:45.129+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:45.359 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:45 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:45.129+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:45.359 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:45 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:45.096+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:45.359 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:45 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:45.096+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:18.142176+0000 front 2024-03-12T03:50:18.142132+0000 (oldest deadline 2024-03-12T03:50:42.142480+0000) 2024-03-12T03:50:45.714 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:45 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:45.355+0000 7f74a42fb700 -1 osd.4 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:11.191076+0000 front 2024-03-12T03:50:11.191278+0000 (oldest deadline 2024-03-12T03:50:36.491131+0000) 2024-03-12T03:50:45.715 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:45 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4[37906]: 2024-03-12T03:50:45.355+0000 7f74a42fb700 -1 osd.4 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:16.491832+0000 front 2024-03-12T03:50:16.491716+0000 (oldest deadline 2024-03-12T03:50:42.391735+0000) 2024-03-12T03:50:46.132 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:45 smithi151 podman[75366]: 2024-03-12 03:50:45.835326436 +0000 UTC m=+5.626264389 container died 13323bdc0ac0cc7eb6d9fd1ac7d08610d4582f01eeac3e6e3dba7c584b299ca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-12T03:50:46.132 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:46 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:46.056+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:46.132 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:46 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:46.056+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:18.142176+0000 front 2024-03-12T03:50:18.142132+0000 (oldest deadline 2024-03-12T03:50:42.142480+0000) 2024-03-12T03:50:46.439 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:46 smithi151 podman[75366]: 2024-03-12 03:50:46.220902849 +0000 UTC m=+6.011840806 container cleanup 13323bdc0ac0cc7eb6d9fd1ac7d08610d4582f01eeac3e6e3dba7c584b299ca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True) 2024-03-12T03:50:46.439 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:46 smithi151 bash[75366]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4 2024-03-12T03:50:46.439 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:46 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:46.300+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:46.440 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:46 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:46.300+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:46.440 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:46 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:46.156+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:46.440 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:46 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:46.156+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:46.714 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:46 smithi151 podman[75384]: 2024-03-12 03:50:46.437254976 +0000 UTC m=+0.603562910 container remove 13323bdc0ac0cc7eb6d9fd1ac7d08610d4582f01eeac3e6e3dba7c584b299ca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:50:47.135 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:47.084+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:47.135 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:47.084+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:18.142176+0000 front 2024-03-12T03:50:18.142132+0000 (oldest deadline 2024-03-12T03:50:42.142480+0000) 2024-03-12T03:50:47.136 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:46 smithi151 podman[75473]: 2024-03-12 03:50:46.785898056 +0000 UTC m=+0.018746771 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:50:47.136 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:46 smithi151 podman[75473]: 2024-03-12 03:50:46.906283595 +0000 UTC m=+0.139132294 container create a75c9b7acd54bc1036746ada8c6190faac60fbb6b43bd0cbaec2e82594c4e4b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-12T03:50:47.136 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:47 smithi151 podman[75473]: 2024-03-12 03:50:47.133843142 +0000 UTC m=+0.366691841 container init a75c9b7acd54bc1036746ada8c6190faac60fbb6b43bd0cbaec2e82594c4e4b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-12T03:50:47.426 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:47.151+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:47.426 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:47.151+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:47.426 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:47 smithi151 podman[75473]: 2024-03-12 03:50:47.137640246 +0000 UTC m=+0.370488949 container start a75c9b7acd54bc1036746ada8c6190faac60fbb6b43bd0cbaec2e82594c4e4b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:50:47.426 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:47 smithi151 podman[75473]: 2024-03-12 03:50:47.22546385 +0000 UTC m=+0.458312565 container attach a75c9b7acd54bc1036746ada8c6190faac60fbb6b43bd0cbaec2e82594c4e4b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4-deactivate, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:50:47.427 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:47.277+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:47.427 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:47 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:47.277+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:47.714 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 12 03:50:47 smithi151 podman[75473]: 2024-03-12 03:50:47.423693601 +0000 UTC m=+0.656542299 container died a75c9b7acd54bc1036746ada8c6190faac60fbb6b43bd0cbaec2e82594c4e4b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-12T03:50:48.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.4.service' 2024-03-12T03:50:48.067 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:50:48.068 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-12T03:50:48.068 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-12T03:50:48.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.5 2024-03-12T03:50:48.289 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:48.161+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:48.289 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:48.161+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:48.290 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:48.119+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:48.290 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:48.119+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:18.142176+0000 front 2024-03-12T03:50:18.142132+0000 (oldest deadline 2024-03-12T03:50:42.142480+0000) 2024-03-12T03:50:48.290 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:48 smithi151 systemd[1]: Stopping Ceph osd.5 for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:50:48.714 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:48.286+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:48.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:48.286+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:48.715 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:48.398+0000 7fca80aac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:50:48.715 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:48.398+0000 7fca80aac700 -1 osd.5 824 *** Got signal Terminated *** 2024-03-12T03:50:48.715 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:48 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:48.398+0000 7fca80aac700 -1 osd.5 824 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:50:49.464 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:49.336+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:49.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:49.336+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:49.465 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:49.131+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:49.465 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:49.131+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:18.142176+0000 front 2024-03-12T03:50:18.142132+0000 (oldest deadline 2024-03-12T03:50:42.142480+0000) 2024-03-12T03:50:49.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:49.194+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:49.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:49 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:49.194+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:50.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:50 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:50.295+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:50.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:50 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:50.295+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:50.465 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:50 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:50.093+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:50.465 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:50 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:50.093+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:18.142176+0000 front 2024-03-12T03:50:18.142132+0000 (oldest deadline 2024-03-12T03:50:42.142480+0000) 2024-03-12T03:50:50.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:50 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:50.234+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:50.466 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:50 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:50.234+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:51.464 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:51 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:51.056+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:51.465 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:51 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:51.056+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:18.142176+0000 front 2024-03-12T03:50:18.142132+0000 (oldest deadline 2024-03-12T03:50:42.142480+0000) 2024-03-12T03:50:51.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:51 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:51.276+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:51.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:51 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:51.276+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:51.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:51 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:51.276+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:28.131119+0000 front 2024-03-12T03:50:28.131002+0000 (oldest deadline 2024-03-12T03:50:51.031005+0000) 2024-03-12T03:50:51.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:51 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:51.260+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:51.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:51 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:51.260+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:52.310 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:52 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:52.042+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:52.310 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:52 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:52.042+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:18.142176+0000 front 2024-03-12T03:50:18.142132+0000 (oldest deadline 2024-03-12T03:50:42.142480+0000) 2024-03-12T03:50:52.310 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:52 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:52.267+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:52.310 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:52 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:52.267+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:52.311 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:52 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:52.267+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:28.131119+0000 front 2024-03-12T03:50:28.131002+0000 (oldest deadline 2024-03-12T03:50:51.031005+0000) 2024-03-12T03:50:52.714 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:52 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:52.307+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:52.714 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:52 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:52.307+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:53.465 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:53 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:53.084+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:09.941217+0000 front 2024-03-12T03:50:09.941364+0000 (oldest deadline 2024-03-12T03:50:35.841163+0000) 2024-03-12T03:50:53.465 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:53 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5[43419]: 2024-03-12T03:50:53.084+0000 7fca7d0a0700 -1 osd.5 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:18.142176+0000 front 2024-03-12T03:50:18.142132+0000 (oldest deadline 2024-03-12T03:50:42.142480+0000) 2024-03-12T03:50:53.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:53 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:53.258+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:53.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:53 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:53.258+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:53.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:53 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:53.258+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:28.131119+0000 front 2024-03-12T03:50:28.131002+0000 (oldest deadline 2024-03-12T03:50:51.031005+0000) 2024-03-12T03:50:53.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:53 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:53.258+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:53.466 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:53 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:53.258+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:54.126 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:53 smithi151 podman[75578]: 2024-03-12 03:50:53.854434241 +0000 UTC m=+5.559275558 container died e01e4b6efcf86e97ea07af28893beb90ad36c71e21aea93f924c98201ed35be9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-2b0101e) 2024-03-12T03:50:54.464 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:54 smithi151 podman[75578]: 2024-03-12 03:50:54.222946698 +0000 UTC m=+5.927788018 container cleanup e01e4b6efcf86e97ea07af28893beb90ad36c71e21aea93f924c98201ed35be9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-12T03:50:54.465 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:54 smithi151 bash[75578]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5 2024-03-12T03:50:54.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:54 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:54.242+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:54.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:54 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:54.242+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:54.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:54 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:54.242+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:50:54.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:54 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:54.303+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:54.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:54 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:54.303+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:54.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:54 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:54.303+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:28.131119+0000 front 2024-03-12T03:50:28.131002+0000 (oldest deadline 2024-03-12T03:50:51.031005+0000) 2024-03-12T03:50:54.843 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:54 smithi151 podman[75606]: 2024-03-12 03:50:54.532966804 +0000 UTC m=+0.676865100 container remove e01e4b6efcf86e97ea07af28893beb90ad36c71e21aea93f924c98201ed35be9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:50:55.162 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:54 smithi151 podman[75699]: 2024-03-12 03:50:54.841953245 +0000 UTC m=+0.114175458 container create 26b5e9cf092e5c51aab4d740a9f24b29a8d9ad1c9dda07e1f604cb3292241a50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, io.buildah.version=1.33.5) 2024-03-12T03:50:55.162 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:54 smithi151 podman[75699]: 2024-03-12 03:50:54.745703461 +0000 UTC m=+0.017925693 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:50:55.162 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:55 smithi151 podman[75699]: 2024-03-12 03:50:55.054012541 +0000 UTC m=+0.326234758 container init 26b5e9cf092e5c51aab4d740a9f24b29a8d9ad1c9dda07e1f604cb3292241a50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-12T03:50:55.163 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:55 smithi151 podman[75699]: 2024-03-12 03:50:55.057715635 +0000 UTC m=+0.329937853 container start 26b5e9cf092e5c51aab4d740a9f24b29a8d9ad1c9dda07e1f604cb3292241a50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-12T03:50:55.163 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:55 smithi151 podman[75699]: 2024-03-12 03:50:55.160817483 +0000 UTC m=+0.433039707 container attach 26b5e9cf092e5c51aab4d740a9f24b29a8d9ad1c9dda07e1f604cb3292241a50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-12T03:50:55.465 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 12 03:50:55 smithi151 podman[75699]: 2024-03-12 03:50:55.3559157 +0000 UTC m=+0.628137915 container died 26b5e9cf092e5c51aab4d740a9f24b29a8d9ad1c9dda07e1f604cb3292241a50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, CEPH_POINT_RELEASE=) 2024-03-12T03:50:55.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:55 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:55.303+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:55.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:55 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:55.303+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:55.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:55 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:55.303+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:28.131119+0000 front 2024-03-12T03:50:28.131002+0000 (oldest deadline 2024-03-12T03:50:51.031005+0000) 2024-03-12T03:50:55.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:55 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:55.268+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:55.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:55 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:55.268+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:55.466 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:55 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:55.268+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:50:56.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.5.service' 2024-03-12T03:50:56.052 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:50:56.052 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-12T03:50:56.052 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-12T03:50:56.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.6 2024-03-12T03:50:56.244 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:56 smithi151 systemd[1]: Stopping Ceph osd.6 for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:50:56.714 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:56 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:56.241+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:56.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:56 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:56.241+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:56.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:56 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:56.241+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:50:56.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:56 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:56.286+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:56.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:56 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:56.286+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:56.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:56 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:56.286+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:28.131119+0000 front 2024-03-12T03:50:28.131002+0000 (oldest deadline 2024-03-12T03:50:51.031005+0000) 2024-03-12T03:50:56.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:56 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:56.375+0000 7f7b96900700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:50:56.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:56 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:56.375+0000 7f7b96900700 -1 osd.6 824 *** Got signal Terminated *** 2024-03-12T03:50:56.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:56 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:56.375+0000 7f7b96900700 -1 osd.6 824 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:50:57.714 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:57.292+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:57.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:57.292+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:57.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:57.292+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:28.131119+0000 front 2024-03-12T03:50:28.131002+0000 (oldest deadline 2024-03-12T03:50:51.031005+0000) 2024-03-12T03:50:57.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:57.260+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:57.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:57.260+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:57.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:57.260+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:50:57.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:57 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:57.260+0000 7f2714745700 -1 osd.7 824 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24463.0:2192 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e824) 2024-03-12T03:50:58.714 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:58.279+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:58.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:58.279+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:58.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:58.279+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:50:58.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:58.279+0000 7f2714745700 -1 osd.7 824 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24463.0:2192 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e824) 2024-03-12T03:50:58.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:58.338+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:58.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:58.338+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:58.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:58 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:58.338+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:28.131119+0000 front 2024-03-12T03:50:28.131002+0000 (oldest deadline 2024-03-12T03:50:51.031005+0000) 2024-03-12T03:50:59.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:59.351+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:50:59.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:59.351+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:50:59.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:50:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:50:59.351+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:28.131119+0000 front 2024-03-12T03:50:28.131002+0000 (oldest deadline 2024-03-12T03:50:51.031005+0000) 2024-03-12T03:50:59.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:59.233+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:50:59.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:59.234+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:50:59.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:59.234+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:50:59.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:50:59 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:50:59.234+0000 7f2714745700 -1 osd.7 824 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24463.0:2192 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e824) 2024-03-12T03:51:00.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:00 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:51:00.392+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.329133+0000 front 2024-03-12T03:50:12.328973+0000 (oldest deadline 2024-03-12T03:50:35.228958+0000) 2024-03-12T03:51:00.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:00 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:51:00.392+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:21.130180+0000 front 2024-03-12T03:50:21.130207+0000 (oldest deadline 2024-03-12T03:50:44.030096+0000) 2024-03-12T03:51:00.465 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:00 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6[49127]: 2024-03-12T03:51:00.392+0000 7f7b8eeec700 -1 osd.6 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:28.131119+0000 front 2024-03-12T03:50:28.131002+0000 (oldest deadline 2024-03-12T03:50:51.031005+0000) 2024-03-12T03:51:00.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:00 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:00.185+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:51:00.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:00 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:00.185+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:51:00.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:00 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:00.185+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:51:01.464 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:01 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:01.201+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:51:01.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:01 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:01.201+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:51:01.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:01 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:01.201+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:51:02.214 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:01 smithi151 podman[75804]: 2024-03-12 03:51:01.937001027 +0000 UTC m=+5.652807817 container died 101cb7ace79e3ea9f8a625faa31a9a5975c2b480b3c0a5f41a37bec787092974 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:51:02.603 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:02 smithi151 podman[75804]: 2024-03-12 03:51:02.308718129 +0000 UTC m=+6.024524912 container cleanup 101cb7ace79e3ea9f8a625faa31a9a5975c2b480b3c0a5f41a37bec787092974 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-12T03:51:02.603 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:02 smithi151 bash[75804]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6 2024-03-12T03:51:02.603 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:02 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:02.225+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:51:02.604 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:02 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:02.225+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:51:02.604 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:02 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:02.225+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:51:02.604 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:02 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:02.225+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6815 osd.3 since back 2024-03-12T03:50:33.342097+0000 front 2024-03-12T03:50:33.342126+0000 (oldest deadline 2024-03-12T03:51:02.142457+0000) 2024-03-12T03:51:02.922 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:02 smithi151 podman[75822]: 2024-03-12 03:51:02.601426976 +0000 UTC m=+0.655344171 container remove 101cb7ace79e3ea9f8a625faa31a9a5975c2b480b3c0a5f41a37bec787092974 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-12T03:51:03.214 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:02 smithi151 podman[75920]: 2024-03-12 03:51:02.823501326 +0000 UTC m=+0.017822134 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:51:03.267 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:02 smithi151 podman[75920]: 2024-03-12 03:51:02.9355738 +0000 UTC m=+0.129894580 container create 31139ad29d7ad5cebefd96666c7601c13913258ea07eea008383f0e92fb1cf4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6-deactivate, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:51:03.267 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:03 smithi151 podman[75920]: 2024-03-12 03:51:03.138224938 +0000 UTC m=+0.332545735 container init 31139ad29d7ad5cebefd96666c7601c13913258ea07eea008383f0e92fb1cf4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:51:03.267 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:03 smithi151 podman[75920]: 2024-03-12 03:51:03.141778504 +0000 UTC m=+0.336099287 container start 31139ad29d7ad5cebefd96666c7601c13913258ea07eea008383f0e92fb1cf4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-12T03:51:03.267 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:03 smithi151 podman[75920]: 2024-03-12 03:51:03.213031338 +0000 UTC m=+0.407352137 container attach 31139ad29d7ad5cebefd96666c7601c13913258ea07eea008383f0e92fb1cf4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6-deactivate, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:51:03.715 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:03 smithi151 podman[75920]: 2024-03-12 03:51:03.434984334 +0000 UTC m=+0.629305125 container died 31139ad29d7ad5cebefd96666c7601c13913258ea07eea008383f0e92fb1cf4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e) 2024-03-12T03:51:03.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:03 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:03.222+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:51:03.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:03 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:03.222+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:51:03.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:03 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:03.222+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:51:03.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:03 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:03.222+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6815 osd.3 since back 2024-03-12T03:50:33.342097+0000 front 2024-03-12T03:50:33.342126+0000 (oldest deadline 2024-03-12T03:51:02.142457+0000) 2024-03-12T03:51:03.981 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 12 03:51:03 smithi151 podman[75947]: 2024-03-12 03:51:03.965997088 +0000 UTC m=+0.523251896 container remove 31139ad29d7ad5cebefd96666c7601c13913258ea07eea008383f0e92fb1cf4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-6-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-12T03:51:03.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.6.service' 2024-03-12T03:51:04.027 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:51:04.027 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-12T03:51:04.027 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-12T03:51:04.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.7 2024-03-12T03:51:04.346 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:04 smithi151 systemd[1]: Stopping Ceph osd.7 for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:51:04.346 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:04 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:04.187+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:51:04.346 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:04 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:04.187+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:51:04.346 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:04 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:04.187+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:51:04.346 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:04 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:04.187+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6815 osd.3 since back 2024-03-12T03:50:33.342097+0000 front 2024-03-12T03:50:33.342126+0000 (oldest deadline 2024-03-12T03:51:02.142457+0000) 2024-03-12T03:51:04.346 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:04 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:04.343+0000 7f2718151700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:51:04.347 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:04 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:04.343+0000 7f2718151700 -1 osd.7 824 *** Got signal Terminated *** 2024-03-12T03:51:04.347 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:04 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:04.343+0000 7f2718151700 -1 osd.7 824 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:51:05.616 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:05 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:05.238+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:51:05.616 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:05 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:05.238+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:51:05.616 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:05 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:05.238+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:51:05.616 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:05 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:05.238+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6815 osd.3 since back 2024-03-12T03:50:33.342097+0000 front 2024-03-12T03:50:33.342126+0000 (oldest deadline 2024-03-12T03:51:02.142457+0000) 2024-03-12T03:51:06.631 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:06 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:06.250+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:51:06.632 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:06 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:06.250+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:51:06.632 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:06 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:06.250+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:51:06.632 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:06 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:06.250+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6815 osd.3 since back 2024-03-12T03:50:33.342097+0000 front 2024-03-12T03:50:33.342126+0000 (oldest deadline 2024-03-12T03:51:02.142457+0000) 2024-03-12T03:51:06.632 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:06 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:06.250+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.151:6802 osd.4 since back 2024-03-12T03:50:42.143059+0000 front 2024-03-12T03:50:42.142998+0000 (oldest deadline 2024-03-12T03:51:06.243003+0000) 2024-03-12T03:51:07.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:07.262+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:51:07.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:07.262+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:51:07.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:07.262+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:51:07.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:07.262+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6815 osd.3 since back 2024-03-12T03:50:33.342097+0000 front 2024-03-12T03:50:33.342126+0000 (oldest deadline 2024-03-12T03:51:02.142457+0000) 2024-03-12T03:51:07.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:07 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:07.262+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.151:6802 osd.4 since back 2024-03-12T03:50:42.143059+0000 front 2024-03-12T03:50:42.142998+0000 (oldest deadline 2024-03-12T03:51:06.243003+0000) 2024-03-12T03:51:08.714 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:08.241+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:51:08.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:08.241+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:51:08.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:08.241+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:51:08.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:08.241+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6815 osd.3 since back 2024-03-12T03:50:33.342097+0000 front 2024-03-12T03:50:33.342126+0000 (oldest deadline 2024-03-12T03:51:02.142457+0000) 2024-03-12T03:51:08.715 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:08 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:08.241+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.151:6802 osd.4 since back 2024-03-12T03:50:42.143059+0000 front 2024-03-12T03:50:42.142998+0000 (oldest deadline 2024-03-12T03:51:06.243003+0000) 2024-03-12T03:51:09.689 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:09.233+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6803 osd.0 since back 2024-03-12T03:50:12.438936+0000 front 2024-03-12T03:50:12.438919+0000 (oldest deadline 2024-03-12T03:50:38.338832+0000) 2024-03-12T03:51:09.689 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:09.233+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6807 osd.1 since back 2024-03-12T03:50:19.439883+0000 front 2024-03-12T03:50:19.439853+0000 (oldest deadline 2024-03-12T03:50:42.339795+0000) 2024-03-12T03:51:09.689 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:09.233+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6811 osd.2 since back 2024-03-12T03:50:29.241619+0000 front 2024-03-12T03:50:29.241674+0000 (oldest deadline 2024-03-12T03:50:53.341595+0000) 2024-03-12T03:51:09.689 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:09.233+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.94:6815 osd.3 since back 2024-03-12T03:50:33.342097+0000 front 2024-03-12T03:50:33.342126+0000 (oldest deadline 2024-03-12T03:51:02.142457+0000) 2024-03-12T03:51:09.689 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:09 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7[55019]: 2024-03-12T03:51:09.233+0000 7f2714745700 -1 osd.7 824 heartbeat_check: no reply from 172.21.15.151:6802 osd.4 since back 2024-03-12T03:50:42.143059+0000 front 2024-03-12T03:50:42.142998+0000 (oldest deadline 2024-03-12T03:51:06.243003+0000) 2024-03-12T03:51:09.954 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:09 smithi151 podman[76026]: 2024-03-12 03:51:09.686757337 +0000 UTC m=+5.446494320 container died a58de13b15690956afcbc41fef2a4da479e5bb601352ae2652d477e69a1237df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-03-12T03:51:10.215 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:10 smithi151 podman[76026]: 2024-03-12 03:51:10.060360445 +0000 UTC m=+5.820097429 container cleanup a58de13b15690956afcbc41fef2a4da479e5bb601352ae2652d477e69a1237df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-12T03:51:10.215 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:10 smithi151 bash[76026]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7 2024-03-12T03:51:10.674 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:10 smithi151 podman[76058]: 2024-03-12 03:51:10.336110328 +0000 UTC m=+0.643157835 container remove a58de13b15690956afcbc41fef2a4da479e5bb601352ae2652d477e69a1237df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-12T03:51:10.964 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:10 smithi151 podman[76149]: 2024-03-12 03:51:10.575747368 +0000 UTC m=+0.017853143 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:51:10.965 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:10 smithi151 podman[76149]: 2024-03-12 03:51:10.695262974 +0000 UTC m=+0.137368753 container create 0956aee60e3c160540f70accbdc3c4732b75fd91fe811dfbe5306d5932341dc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7-deactivate, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-12T03:51:10.965 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:10 smithi151 podman[76149]: 2024-03-12 03:51:10.889123886 +0000 UTC m=+0.331229675 container init 0956aee60e3c160540f70accbdc3c4732b75fd91fe811dfbe5306d5932341dc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-12T03:51:10.965 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:10 smithi151 podman[76149]: 2024-03-12 03:51:10.892173205 +0000 UTC m=+0.334278983 container start 0956aee60e3c160540f70accbdc3c4732b75fd91fe811dfbe5306d5932341dc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-12T03:51:11.464 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:10 smithi151 podman[76149]: 2024-03-12 03:51:10.972276441 +0000 UTC m=+0.414382231 container attach 0956aee60e3c160540f70accbdc3c4732b75fd91fe811dfbe5306d5932341dc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-12T03:51:11.465 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:11 smithi151 podman[76149]: 2024-03-12 03:51:11.181110706 +0000 UTC m=+0.623216484 container died 0956aee60e3c160540f70accbdc3c4732b75fd91fe811dfbe5306d5932341dc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-12T03:51:11.729 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 12 03:51:11 smithi151 podman[76174]: 2024-03-12 03:51:11.725271106 +0000 UTC m=+0.538586830 container remove 0956aee60e3c160540f70accbdc3c4732b75fd91fe811dfbe5306d5932341dc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e) 2024-03-12T03:51:11.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@osd.7.service' 2024-03-12T03:51:11.790 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:51:11.790 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-12T03:51:11.790 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-12T03:51:11.790 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-12T03:51:12.083 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:51:11 smithi094 systemd[1]: Stopping Ceph rgw.foo.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:51:12.459 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:51:12 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-rgw-foo-a[67582]: 2024-03-12T03:51:12.080+0000 7f8d3e735700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:51:12.459 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:51:12 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-rgw-foo-a[67582]: 2024-03-12T03:51:12.080+0000 7f8d4cfadac0 -1 shutting down 2024-03-12T03:51:22.376 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:51:22 smithi094 bash[157133]: time="2024-03-12T03:51:22Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-964142e8-e01d-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-12T03:51:22.376 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:51:22 smithi094 podman[157133]: 2024-03-12 03:51:22.109203897 +0000 UTC m=+10.132188308 container died 6b571fe46f3e20ba9e07b75adbe95e9d466797351a6fcf0e4f490e0890cdb618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-rgw-foo-a, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:51:22.709 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:51:22 smithi094 podman[157133]: 2024-03-12 03:51:22.489990782 +0000 UTC m=+10.512975188 container cleanup 6b571fe46f3e20ba9e07b75adbe95e9d466797351a6fcf0e4f490e0890cdb618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-12T03:51:22.709 INFO:journalctl@ceph.rgw.foo.a.smithi094.stdout:Mar 12 03:51:22 smithi094 bash[157133]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-12T03:51:22.913 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-12T03:51:22.952 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:51:22.952 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-12T03:51:22.952 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-12T03:51:22.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-964142e8-e01d-11ee-95c8-87774f69a715@prometheus.a 2024-03-12T03:51:23.262 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 systemd[1]: Stopping Ceph prometheus.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:51:23.529 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:51:23.260Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:51:23.529 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:51:23.260Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-12T03:51:23.530 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:51:23.260Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-12T03:51:23.530 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:51:23.260Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-12T03:51:23.530 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:51:23.260Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-12T03:51:23.530 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:51:23.260Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-12T03:51:23.530 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:51:23.260Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-12T03:51:23.530 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:51:23.261Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-12T03:51:23.530 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:51:23.261Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-12T03:51:23.530 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:51:23.261Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-12T03:51:23.530 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:51:23.261Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-12T03:51:23.530 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a[64874]: ts=2024-03-12T03:51:23.261Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-12T03:51:23.530 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 podman[76255]: 2024-03-12 03:51:23.270757521 +0000 UTC m=+0.115550842 container died af733172183c6dcbacc62caedbf96b30a6ce278a8f0354265f18d4c2c5e2fe6b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:51:23.904 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 podman[76255]: 2024-03-12 03:51:23.626878123 +0000 UTC m=+0.471671433 container cleanup af733172183c6dcbacc62caedbf96b30a6ce278a8f0354265f18d4c2c5e2fe6b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:51:23.904 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 12 03:51:23 smithi151 bash[76255]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-prometheus-a 2024-03-12T03:51:24.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-964142e8-e01d-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-12T03:51:24.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:51:24.092 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-12T03:51:24.092 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm rm-cluster --fsid 964142e8-e01d-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T03:51:24.283 INFO:teuthology.orchestra.run.smithi094.stdout:Deleting cluster with fsid: 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:51:25.959 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:51:25 smithi094 systemd[1]: Stopping Ceph alertmanager.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:51:26.399 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:51:26 smithi094 ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a[77353]: ts=2024-03-12T03:51:26.086Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:51:26.399 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:51:26 smithi094 podman[157638]: 2024-03-12 03:51:26.127700706 +0000 UTC m=+0.173537111 container died e3549d8f9563f70d4c72f1abec3831ab16f696586d0f814b6c7bd7f0f31402f0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:51:26.709 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:51:26 smithi094 podman[157638]: 2024-03-12 03:51:26.496387688 +0000 UTC m=+0.542224156 container cleanup e3549d8f9563f70d4c72f1abec3831ab16f696586d0f814b6c7bd7f0f31402f0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:51:26.709 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:51:26 smithi094 bash[157638]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a 2024-03-12T03:51:27.150 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:51:26 smithi094 podman[157652]: 2024-03-12 03:51:26.752017993 +0000 UTC m=+0.619412968 container remove e3549d8f9563f70d4c72f1abec3831ab16f696586d0f814b6c7bd7f0f31402f0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:51:27.150 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:51:26 smithi094 podman[157652]: 2024-03-12 03:51:26.82791544 +0000 UTC m=+0.695310416 volume remove c83964b83dc7bb90ab63cc95e5747f5b3c0608f16749531d06ac1e248d19f983 2024-03-12T03:51:27.150 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:51:26 smithi094 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-12T03:51:27.150 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:51:26 smithi094 systemd[1]: Stopped Ceph alertmanager.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:51:27.150 INFO:journalctl@ceph.alertmanager.a.smithi094.stdout:Mar 12 03:51:26 smithi094 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 2.343s CPU time. 2024-03-12T03:51:27.459 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:51:27 smithi094 systemd[1]: Stopping Ceph node-exporter.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:51:27.459 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:51:27 smithi094 podman[157764]: 2024-03-12 03:51:27.391273974 +0000 UTC m=+0.109516546 container died f3063f7b48c78f1a1d78eb226f23a0a0884adcbe7f4495e91b91da3c08c1c65d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:51:27.926 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:51:27 smithi094 podman[157764]: 2024-03-12 03:51:27.722966332 +0000 UTC m=+0.441208903 container cleanup f3063f7b48c78f1a1d78eb226f23a0a0884adcbe7f4495e91b91da3c08c1c65d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:51:27.926 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:51:27 smithi094 bash[157764]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a 2024-03-12T03:51:28.209 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:51:27 smithi094 podman[157778]: 2024-03-12 03:51:27.92462303 +0000 UTC m=+0.534473200 container remove f3063f7b48c78f1a1d78eb226f23a0a0884adcbe7f4495e91b91da3c08c1c65d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:51:28.209 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:51:27 smithi094 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:51:28.210 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:51:28 smithi094 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-12T03:51:28.210 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:51:28 smithi094 systemd[1]: Stopped Ceph node-exporter.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:51:28.210 INFO:journalctl@ceph.node-exporter.a.smithi094.stdout:Mar 12 03:51:28 smithi094 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 5.063s CPU time. 2024-03-12T03:51:28.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm rm-cluster --fsid 964142e8-e01d-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T03:51:28.939 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:51:30.408 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:51:30 smithi151 systemd[1]: Stopping Ceph iscsi.iscsi.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:51:30.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:51:30 smithi151 ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a[60112]: debug Shutdown received 2024-03-12T03:51:40.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:51:40 smithi151 bash[76669]: time="2024-03-12T03:51:40Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-12T03:51:40.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:51:40 smithi151 podman[76669]: 2024-03-12 03:51:40.420943333 +0000 UTC m=+10.117879671 container died d08b29abd1003b96824f38a0c4dfe878a098451967303ed8c37496be3fd361bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-2b0101e) 2024-03-12T03:51:40.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:51:40 smithi151 podman[76669]: 2024-03-12 03:51:40.769360422 +0000 UTC m=+10.466296774 container cleanup d08b29abd1003b96824f38a0c4dfe878a098451967303ed8c37496be3fd361bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-12T03:51:40.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:51:40 smithi151 bash[76669]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a 2024-03-12T03:51:41.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:51:41 smithi151 podman[76684]: 2024-03-12 03:51:41.062136194 +0000 UTC m=+0.641938322 container remove d08b29abd1003b96824f38a0c4dfe878a098451967303ed8c37496be3fd361bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-iscsi-iscsi-a, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:51:41.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:51:41 smithi151 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-12T03:51:41.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:51:41 smithi151 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-12T03:51:41.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:51:41 smithi151 systemd[1]: Stopped Ceph iscsi.iscsi.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:51:41.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 12 03:51:41 smithi151 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Consumed 4.341s CPU time. 2024-03-12T03:51:42.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:51:42 smithi151 systemd[1]: Stopping Ceph grafana.a for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:51:43.295 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:51:42 smithi151 podman[76956]: 2024-03-12 03:51:42.962616915 +0000 UTC m=+0.172608206 container died 3bde7f9e682038bcf2c8bdcc9cacec45b44c066fef34063eee5d4afb390ee570 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a, architecture=x86_64, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, maintainer=Paul Cuzner , version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, com.redhat.component=ubi8-container) 2024-03-12T03:51:43.295 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:51:43 smithi151 podman[76956]: 2024-03-12 03:51:43.293456689 +0000 UTC m=+0.503447988 container cleanup 3bde7f9e682038bcf2c8bdcc9cacec45b44c066fef34063eee5d4afb390ee570 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a, vendor=Red Hat, Inc., io.buildah.version=1.32.0, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, version=8.5, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, io.openshift.expose-services=, io.openshift.tags=base rhel8, release=239.1651231664) 2024-03-12T03:51:43.684 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:51:43 smithi151 bash[76956]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a 2024-03-12T03:51:43.684 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:51:43 smithi151 podman[76972]: 2024-03-12 03:51:43.486200452 +0000 UTC m=+0.526229183 container remove 3bde7f9e682038bcf2c8bdcc9cacec45b44c066fef34063eee5d4afb390ee570 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-grafana-a, vcs-type=git, name=ubi8, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=239.1651231664, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, version=8.5, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , com.redhat.component=ubi8-container) 2024-03-12T03:51:43.684 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:51:43 smithi151 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:51:43.964 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:51:43 smithi151 systemd[1]: Stopping Ceph node-exporter.b for 964142e8-e01d-11ee-95c8-87774f69a715... 2024-03-12T03:51:43.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:51:43 smithi151 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-12T03:51:43.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:51:43 smithi151 systemd[1]: Stopped Ceph grafana.a for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:51:43.965 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 12 03:51:43 smithi151 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@grafana.a.service: Consumed 18.098s CPU time. 2024-03-12T03:51:44.425 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:51:44 smithi151 podman[77084]: 2024-03-12 03:51:44.149651961 +0000 UTC m=+0.114332058 container died a6926598bdcce3ec6dea19d1bb8b8c92d760615572f91593da30bc72d1d628c6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:51:44.714 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:51:44 smithi151 podman[77084]: 2024-03-12 03:51:44.531081751 +0000 UTC m=+0.495761856 container cleanup a6926598bdcce3ec6dea19d1bb8b8c92d760615572f91593da30bc72d1d628c6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:51:44.715 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:51:44 smithi151 bash[77084]: ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b 2024-03-12T03:51:45.102 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:51:44 smithi151 podman[77098]: 2024-03-12 03:51:44.798880096 +0000 UTC m=+0.649869487 container remove a6926598bdcce3ec6dea19d1bb8b8c92d760615572f91593da30bc72d1d628c6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-964142e8-e01d-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:51:45.102 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:51:44 smithi151 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:51:45.102 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:51:44 smithi151 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-12T03:51:45.102 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:51:44 smithi151 systemd[1]: Stopped Ceph node-exporter.b for 964142e8-e01d-11ee-95c8-87774f69a715. 2024-03-12T03:51:45.102 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 12 03:51:44 smithi151 systemd[1]: ceph-964142e8-e01d-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 5.041s CPU time. 2024-03-12T03:51:45.587 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:51:45.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:51:45.652 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-12T03:51:45.653 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854/remote/smithi094/crash 2024-03-12T03:51:45.654 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/crash -- . 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi094.stderr:tar: /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi094.stderr:tar: Error is not recoverable: exiting now 2024-03-12T03:51:45.709 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854/remote/smithi151/crash 2024-03-12T03:51:45.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/crash -- . 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/964142e8-e01d-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2024-03-12T03:51:45.752 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-12T03:51:45.753 DEBUG:teuthology.orchestra.run.smithi094:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-12T03:51:45.785 INFO:teuthology.orchestra.run.smithi094.stdout:2024-03-12T03:20:00.000255+0000 mon.a (mon.0) 2080 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-12T03:51:45.785 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-12T03:51:45.785 DEBUG:teuthology.orchestra.run.smithi094:> sudo egrep '\[SEC\]' /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-12T03:51:45.857 DEBUG:teuthology.orchestra.run.smithi094:> sudo egrep '\[ERR\]' /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-12T03:51:45.929 DEBUG:teuthology.orchestra.run.smithi094:> sudo egrep '\[WRN\]' /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-12T03:51:46.000 INFO:teuthology.orchestra.run.smithi094.stdout:2024-03-12T03:20:00.000255+0000 mon.a (mon.0) 2080 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-12T03:51:46.000 INFO:tasks.cephadm:Compressing logs... 2024-03-12T03:51:46.000 DEBUG:teuthology.orchestra.run.smithi094:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T03:51:46.044 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T03:51:46.068 INFO:teuthology.orchestra.run.smithi094.stderr:find: ‘/var/log/rbd-target-api’gzip -5: No such file or directory --verbose 2024-03-12T03:51:46.069 INFO:teuthology.orchestra.run.smithi094.stderr: -- /var/log/ceph/cephadm.log 2024-03-12T03:51:46.069 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-12T03:51:46.069 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T03:51:46.069 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-12T03:51:46.069 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-12T03:51:46.069 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-12T03:51:46.069 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-12T03:51:46.069 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-12T03:51:46.072 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log: No such file or directory 2024-03-12T03:51:46.072 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-12T03:51:46.072 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-12T03:51:46.072 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T03:51:46.072 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-12T03:51:46.072 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.6.log: /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-12T03:51:46.072 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-12T03:51:46.077 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-12T03:51:46.077 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/tcmu-runner.log 2024-03-12T03:51:46.077 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T03:51:46.077 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/tcmu-runner.log: 67.6% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/tcmu-runner.log.gz 2024-03-12T03:51:46.078 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.log 2024-03-12T03:51:46.078 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.cephadm.log: 81.6% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T03:51:46.078 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T03:51:46.078 INFO:teuthology.orchestra.run.smithi151.stderr: 84.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T03:51:46.078 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-12T03:51:46.079 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mgr.y.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T03:51:46.079 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T03:51:46.082 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-volume.log: 87.8% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T03:51:46.084 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mon.b.log: 92.2% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T03:51:46.085 INFO:teuthology.orchestra.run.smithi151.stderr: 92.4% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-12T03:51:46.088 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.0.log: 95.2% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T03:51:46.088 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.log 2024-03-12T03:51:46.090 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.cephadm.log: 89.0% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T03:51:46.090 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-12T03:51:46.091 INFO:teuthology.orchestra.run.smithi151.stderr: 90.8% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T03:51:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T03:51:46.103 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.2.log: 93.2% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T03:51:46.108 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-12T03:51:46.149 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T03:51:46.153 INFO:teuthology.orchestra.run.smithi094.stderr: 93.4% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-12T03:51:46.615 INFO:teuthology.orchestra.run.smithi094.stderr: 90.1% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-12T03:51:47.942 INFO:teuthology.orchestra.run.smithi094.stderr: 92.0% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-12T03:51:48.180 INFO:teuthology.orchestra.run.smithi151.stderr: 92.5% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-12T03:51:48.750 INFO:teuthology.orchestra.run.smithi094.stderr: 91.6% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-12T03:51:53.656 INFO:teuthology.orchestra.run.smithi151.stderr: 93.8% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-12T03:51:53.997 INFO:teuthology.orchestra.run.smithi151.stderr: 93.8% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-12T03:51:54.484 INFO:teuthology.orchestra.run.smithi151.stderr: 93.7% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-12T03:51:54.855 INFO:teuthology.orchestra.run.smithi094.stderr: 93.9% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-12T03:51:55.072 INFO:teuthology.orchestra.run.smithi094.stderr: 93.8% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-12T03:51:55.118 INFO:teuthology.orchestra.run.smithi094.stderr: 93.9% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-12T03:51:55.137 INFO:teuthology.orchestra.run.smithi094.stderr: 93.9% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-12T03:51:55.139 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-12T03:51:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:real 0m9.085s 2024-03-12T03:51:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:user 0m39.612s 2024-03-12T03:51:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:sys 0m1.323s 2024-03-12T03:51:55.193 INFO:teuthology.orchestra.run.smithi151.stderr: 93.9% -- replaced with /var/log/ceph/964142e8-e01d-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-12T03:51:55.195 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-12T03:51:55.195 INFO:teuthology.orchestra.run.smithi151.stderr:real 0m9.136s 2024-03-12T03:51:55.195 INFO:teuthology.orchestra.run.smithi151.stderr:user 0m33.962s 2024-03-12T03:51:55.195 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m1.191s 2024-03-12T03:51:55.195 INFO:tasks.cephadm:Archiving logs... 2024-03-12T03:51:55.196 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854/remote/smithi094/log 2024-03-12T03:51:55.197 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T03:52:00.669 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854/remote/smithi151/log 2024-03-12T03:52:00.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T03:52:05.737 INFO:tasks.cephadm:Removing cluster... 2024-03-12T03:52:05.738 DEBUG:teuthology.orchestra.run.smithi094:> sudo cephadm rm-cluster --fsid 964142e8-e01d-11ee-95c8-87774f69a715 --force 2024-03-12T03:52:05.932 INFO:teuthology.orchestra.run.smithi094.stdout:Deleting cluster with fsid: 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:52:06.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo cephadm rm-cluster --fsid 964142e8-e01d-11ee-95c8-87774f69a715 --force 2024-03-12T03:52:06.420 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: 964142e8-e01d-11ee-95c8-87774f69a715 2024-03-12T03:52:06.712 INFO:tasks.cephadm:Teardown complete 2024-03-12T03:52:06.713 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-12T03:52:06.722 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-12T03:52:06.722 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-12T03:52:06.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-12T03:52:06.807 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-12T03:52:06.817 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-12T03:52:06.817 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T03:52:06.820 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T03:52:06.831 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-03-12T03:52:06.833 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-03-12T03:52:06.835 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T03:52:06.836 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-03-12T03:52:06.836 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 -395us[ -395us] +/- 108ms 2024-03-12T03:52:06.836 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -1131us[-1073us] +/- 51ms 2024-03-12T03:52:06.836 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +1882us[+1882us] +/- 73ms 2024-03-12T03:52:06.836 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:52:06.837 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T03:52:06.837 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-03-12T03:52:06.837 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 -418us[ -378us] +/- 108ms 2024-03-12T03:52:06.837 INFO:teuthology.orchestra.run.smithi151.stdout:^* hv02.front.sepia.ceph.com 3 7 377 36 -1149us[-1109us] +/- 51ms 2024-03-12T03:52:06.837 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +1871us[+1911us] +/- 73ms 2024-03-12T03:52:06.837 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:52:06.838 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-12T03:52:06.848 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-12T03:52:06.848 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-12T03:52:06.858 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T03:52:06.953 DEBUG:teuthology.orchestra.run.smithi151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T03:52:07.027 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T03:52:07.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:07.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T03:52:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:07.088 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-12T03:52:07.099 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-12T03:52:07.114 INFO:teuthology.task.internal:Duration was 3140.206230 seconds 2024-03-12T03:52:07.114 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-12T03:52:07.124 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-12T03:52:07.124 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T03:52:07.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T03:52:07.167 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T03:52:07.174 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T03:52:07.503 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-12T03:52:07.503 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2024-03-12T03:52:07.503 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T03:52:07.553 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2024-03-12T03:52:07.554 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T03:52:07.582 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-12T03:52:07.582 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T03:52:07.596 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T03:52:07.650 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-12T03:52:07.651 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T03:52:07.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T03:52:08.092 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-12T03:52:08.103 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-12T03:52:08.103 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T03:52:08.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T03:52:08.158 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-12T03:52:08.168 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T03:52:08.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T03:52:08.195 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2024-03-12T03:52:08.226 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2024-03-12T03:52:08.247 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T03:52:08.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:08.265 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T03:52:08.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:08.302 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-12T03:52:08.312 INFO:teuthology.task.internal:Transferring archived files... 2024-03-12T03:52:08.313 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854/remote/smithi094 2024-03-12T03:52:08.313 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T03:52:08.379 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854/remote/smithi151 2024-03-12T03:52:08.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T03:52:08.419 INFO:teuthology.task.internal:Removing archive directory... 2024-03-12T03:52:08.419 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T03:52:08.422 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T03:52:08.468 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-12T03:52:08.481 INFO:teuthology.task.internal:Not uploading archives. 2024-03-12T03:52:08.481 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-12T03:52:08.492 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-12T03:52:08.492 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T03:52:08.494 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T03:52:08.506 INFO:teuthology.orchestra.run.smithi094.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 03:52 /home/ubuntu/cephtest 2024-03-12T03:52:08.522 INFO:teuthology.orchestra.run.smithi151.stdout: 265678 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 03:52 /home/ubuntu/cephtest 2024-03-12T03:52:08.524 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-12T03:52:08.536 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-12T03:52:08.560 INFO:teuthology.nuke:Checking targets against current locks 2024-03-12T03:52:08.587 DEBUG:teuthology.nuke:shortname: smithi094 2024-03-12T03:52:08.587 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T03:52:08.613 DEBUG:teuthology.nuke:shortname: smithi151 2024-03-12T03:52:08.614 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T03:52:08.617 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:50:43.752653', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOZmR8x6IzoXKKffGo75V8AXIimC40GWHexbpHKPBZIRfJOBXq5tBOFFWc/69OPlPjzeNpehFVhuncEkmAYMDI='} 2024-03-12T03:52:08.642 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:50:43.754495', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPTqnZGitYpcpb572mNbfZYAVXyeWniw3x8eD+/VyOZ7qdbAZyg3qwkHUkR1NHCpyNiYQwwNlxYvfCAWW6gGV54='} 2024-03-12T03:52:08.672 INFO:teuthology.orchestra.console.smithi094:Power off 2024-03-12T03:52:08.672 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T03:52:08.699 DEBUG:teuthology.orchestra.console.smithi094:power off output: Chassis Power Control: Down/Off 2024-03-12T03:52:08.700 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T03:52:08.714 INFO:teuthology.orchestra.console.smithi151:Power off 2024-03-12T03:52:08.715 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T03:52:08.725 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-03-12T03:52:08.739 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2024-03-12T03:52:08.739 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T03:52:08.764 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-12T03:52:12.726 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T03:52:12.765 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T03:52:12.836 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-03-12T03:52:12.875 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-12T03:52:16.839 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T03:52:16.876 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T03:52:16.951 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is off 2024-03-12T03:52:16.985 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-12T03:52:17.053 INFO:teuthology.orchestra.console.smithi094:Power off completed 2024-03-12T03:52:20.986 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T03:52:21.095 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2024-03-12T03:52:21.195 INFO:teuthology.orchestra.console.smithi151:Power off completed 2024-03-12T03:52:21.301 INFO:teuthology.run:Summary data: description: orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} duration: 3140.206230401993 failure_reason: '"2024-03-12T03:20:00.000255+0000 mon.a (mon.0) 2080 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-12T03:52:21.301 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T03:52:21.385 INFO:teuthology.run:FAIL